:W&Bá¾&OúºM ioojqu7mª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0‹Á7ü‚" 38ma6nitppy1 ioojqu7mŠØ ioojqu7mjuju11-uc-san-diegoSPAtop"› _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.04290625533447937 epochs‚10  hidden_dim‚256 ( learning_rate‚0.00038751409405277527  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚14  num_encoder_layers‚6Bjjtttt_vanilla_ioojqu7mb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-2-spfhcŠ ·ÈåÔ èÝÅš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64P-7 ioojqu7m&  _runtime‚0  _wandb.runtime‚0ÿU°øjÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ¶ÈåÔøËìå*-of*=/data/spatop/options_files/sweep_tttt_jj/config_ioojqu7m.json*-l* /scratch/logs*-n*jjtttt_vanilla_ioojqu7m2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-2-spfhcr/opt/conda/bin/pythonº htpkk5v3hjos9m11fks5n5dppchdvkb1ÏîæÈÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ þœ³"ª€àäÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-1512559d-7724-700d-39a3-dab660c9a52eÊ13.3º htpkk5v3hjos9m11fks5n5dppchdvkb1µª½Ç2  requirements.txt9¹}IW ioojqu7mZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64cÑ Y ioojqu7mZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64VW²*  ioojqu7mb trainer/global_step2#L6;\  ioojqu7mZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64ùÛÆ’/  ioojqu7mb*"trainer/global_step2׫J  ioojqu7mj9 ¸ÈåÔø­Š*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] õõ)W“  ioojqu7mj  ¸ÈåÔØ‘ÌÉî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 7.9 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 151 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘  Z\J\ ioojqu7mjÊ ¸ÈåÔèÓÍÊ·Trainable params: 162 M Non-trainable params: 86 Total params: 162 M Total estimated model params size (MB): 651.580 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 hÄÄq ioojqu7mj` ¸ÈåÔÐûìÊPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). +Ç{ ioojqu7mjj ¸ÈåÔà…ËZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). R@fy ioojqu7mjh ¸ÈåÔ€€ËXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ÛU € ioojqu7mjo ¸ÈåÔÐñ™Ë_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ®rBz ioojqu7mji ¸ÈåÔø„«ËYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). qûSt ioojqu7mjc ¸ÈåÔØͲËSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). Y© Éq ioojqu7mj` ¸ÈåÔð§¼ËPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). mVGt ioojqu7mjc ¸ÈåÔø¹ÃËSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ¶µô&n ioojqu7mj] ¸ÈåÔØùÍËMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). :ÚÅm ioojqu7mj\ ¸ÈåÔ°ÙËLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ½ƒ Zb ioojqu7m*Ð  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"w5ƒë ioojqu7mjÙ ¸ÈåÔø˜¿ÐÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ãŸÕN ioojqu7mj¼ ¸ÈåÔ¨ÜÍЫ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ùÍ®±ë ioojqu7mjÙ »ÈåÔ¸Š»ÌÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. —zåeP ioojqu7mj¾ »ÈåÔÐÏËÌ­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ßQÂ’‚(:ˆ ÆÈåÔ÷¢É gpu.0.gpu‚60.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚28.173904418945312* gpu.0.memoryAllocatedBytes‚ 24201199616 gpu.0.temp‚40.0 gpu.0.powerWatts‚263.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ’õ,ø‚(:î ÆÈåÔðô¥Ñ network.sent‚34370% disk./.usageGB‚1100.8063049316406 disk.rbd3.out‚0 disk.dm-0.in‚ 11.73046875 disk.rbd1.in‚0 disk.dm-0.out‚99.74462890625# memory_percent‚91.5212869644165" proc.memory.rssMB‚ 2888.6953125* proc.memory.percent‚17.631196975708008 network.recv‚40947 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚0.71875 disk.sda2.in‚0 disk.nvme0n1.in‚ 11.73046875 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 11.73046875% disk.nvme0c0n1.out‚99.74462890625 disk.dm-1.out‚0.71875 disk.drbd1777.out‚0( disk./.usagePercent‚73.8760347619728 disk.drbd1777.in‚0 disk.rbd1.out‚0) proc.memory.availableMB‚ 1389.15234375 cpu‚24.90711419486472 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚99.74462890625 disk.rbd0.in‚0 disk.rbd2.out‚ 1.01171875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.out‚0.71875îúß9’ ‚(:ˆ ÕÈåÔ­ú‚É gpu.0.gpu‚87.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.18122863769531* gpu.0.memoryAllocatedBytes‚ 24207491072 gpu.0.temp‚41.0 gpu.0.powerWatts‚153.087' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡n}!‚(:ö ÕÈåÔâدË disk.sda1.out‚0 disk.rbd2.out‚ 2.1953125 disk.rbd3.out‚0) proc.memory.availableMB‚ 1386.82421875 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 11.73046875 disk.sda3.in‚0 disk.sda3.out‚ 1.3828125! disk.dm-0.out‚100.12353515625( disk./.usagePercent‚73.8760370660406$ disk./.usageGB‚1100.806339263916 disk.drbd1777.out‚0 network.recv‚59709 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 1.3828125 network.sent‚52043 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚100.12353515625# proc.memory.rssMB‚ 2888.73828125 proc.cpu.threads‚17 disk.rbd1.out‚0 disk.dm-1.out‚ 1.3828125) proc.memory.percent‚17.63145923614502! disk.nvme0c0n1.in‚ 11.73046875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚100.12353515625$ memory_percent‚91.53549671173096 disk.sda2.in‚0 disk.dm-0.in‚ 11.73046875 cpu‚24.94583112063416—;Äa3"b/ lr-AdamW/pg1"trainer/global_step2:JP<Ñsˆ3#b/ lr-AdamW/pg2"trainer/global_step2:JPØ#ðû-$) _wandbruntime‚43  _runtime‚43Aa»Á%¼ ' lr-AdamW/pg1‚1.5705699428110826e-05 ' lr-AdamW/pg2‚1.5705699428110826e-05 trainer/global_step‚49 $ _timestamp‚1.788437602997727e+09 _runtime‚ 43.042505081 _step‚0ØeSÚÃ&¾ _step‚0 ' lr-AdamW/pg1‚1.5705699428110826e-05 ' lr-AdamW/pg2‚1.5705699428110826e-05 trainer/global_step‚49 $ _timestamp‚1.788437602997727e+09 _runtime‚ 43.042505081Ó)·n'bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP´Œi(be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP¸×£¿h)bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP}ùq*bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP†œÏ!s+bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP®Òìg,bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP—Üh4f-bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP¥SMFf.bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP E^ýc/b_ ¢Kg:bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPÊÉ n;bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPÌÜ8]gbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPñ¼xün?bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP-Öêg@bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPû¤&nfAbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPx9ÄiBbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP{pŽ&rCbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPLDqDbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPBVlrEbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPõn\–^FbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPJ:f_Gb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP~âíNjHbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPŽ²îdIb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP÷:ï=lJbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP¾hº³cKb_ DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPßÖÑ(hnbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP´;‹Âsobo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPýëÇipbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP+hZDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPc8Åóoxbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP-i`)gybc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPÔK —^zbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPM†¥És{bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPt›Ÿ&g|bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPãú'ƒf}bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPZmwyp~bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP ÇnOobk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP­$¿Åt€bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP­ Ùû`b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP7ùŽf‚ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP±Œ‹“oƒbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPÑ -Ìm„bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPïé›r…bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÏxg†bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPÔ¬ô‹j‡be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPö¦kˆbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP–Û7âk‰bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP€$½ÌqŠbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPøÌ1 k‹bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP|+ÓciŒbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP D›‘fba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP ~ÔcnŽbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP AV8sbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP†í•sbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPYiØân‘bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPЩ2Å^’bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP&Ø–l“bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP#s¤j”be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP?¥J™i•bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP¬¬ j–be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP-}Ž5o—bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPà f˜ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPÅ‹ßo™bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP e:`qšbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPcPão›bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPLqçnœbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPõCùrhbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP츋£ržbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP‡a‰nŸbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPfKîn bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPt5w¡br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP[¦b.¢) _wandbruntime‚43  _runtime‚43…Dû£ŸO YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5432542865408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5350743071744 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1569 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚910243 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5432542865408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2607545344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 89947922458 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5432569256960 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5350717643776 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 92546199578 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚719357 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4802431784448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1569 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚910243 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 92699043840 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5522642466816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚765 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚462188 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4802358583808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5525268300800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚340427 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3290431488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787573248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5522642466816 trainer/global_step‚49 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 288645632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23660068864 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5350743071744 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1569 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚910243 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5432542865408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2607545344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 89947922458 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5432569256960 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1364 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5350717643776 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 92546199578 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚719357 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4802431784448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1569 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚910243 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 92699043840 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5522642466816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚765 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚462188 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4802358583808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5525268300800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚340427 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3290431488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787573248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5522642466816 trainer/global_step‚49 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 288645632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23660068864 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPHŽm®bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPAಂo¯bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPV®*ya°b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPë–*œf±ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPÓˆKr²bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP´–ÝÚl³bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP€)ƒ³o´bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPþt¯rµbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPSKf¶ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÊ ÎÐu·bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPî" d¸b_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPâ[S;_ÓbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP¹ÃoÔbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÔÐhÕbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP³òxdÖb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP²*;XfØba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPD=?^jÙbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPÁ2ÔeÚb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP\;Ã7_ÛbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP*ÁigÜbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP‡Ãc•gÝbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÒmŽ’pÞbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPXºEbßb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP^\Û/càb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPD€ñÃmábh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPlµd ]âbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP‚|Ûkãbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPwÔ6häbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP”V?*fåba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP=¢qæbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPo]›«]çbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPòcFVèbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPéËOiébd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP“¡ keêb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP˜îr€hëbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPì]¹‡lìbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÉõeíb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPÌñlîbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPéL-mïbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPDÇ#êuðbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP5$NÂiñbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPu—…eòb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP4q0Ïróbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPœ"cRjôbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPvYÛghõbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP«¯žcgöbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP®Eö7h÷bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPLtÒkøbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPÞQ6qùbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÙßjbúb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPÞ‚kÓaûb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPÛG¨mübh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPZñÍÀhýbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPån®wþbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP)¬ rÿbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPôSj+g€bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP)lbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP6Ém‚bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPmDßeƒb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP¢uÚk„bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPI)¦Ão…bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP)lUm†bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP8=Cp‡bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP˜ ë‘jˆbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPâ„ae‰b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP¸°fŠba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP¯*ƒ\k‹bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPdE°ÁkŒbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPáW:knbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPu+ÑwŽbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP³4!¹jbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP}^•ümbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPªÝc‘b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPw¹¬q’bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPÇÂ`“b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPckŽ-^”bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPëÀÂzg•bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPÒŸ d–b_ hžbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP2ÅpŸbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPkEƒ€. ) _wandbruntime‚43  _runtime‚43Ø×&ËO¡ªM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3290431488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 91931930112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5540482570752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4897287667200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5540508962304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚818472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1569 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚928477 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5548836083174 _runtime‚ 43.781179974 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5632414500864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 91931930112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚471384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1569 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5635040348160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚765 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚733587 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚347091 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23660068864 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94531385856 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚818917 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23660068864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4766737577472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 289557504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1364 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2607545344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚49 PEDeviceStatsMonitor.on_train_batch_enÉÒV`ad/allocation.small_pool.allocated‚947039 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4897360854528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚2 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5635040348160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 130623277056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2643 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚347533 % _timestamp‚1.7884376037361538e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚471381 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3290431488 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2643 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 94375542454 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3290431488 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94531385856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5540508962304 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚471381 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3290431488 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2643 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 94375542454 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3290431488 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94531385856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5540508962304 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5540482570752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4897287667200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5540508962304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚818472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1569 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚928477 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5548836083174 _runtime‚ 43.781179974 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5632414500864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 91931930112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚471384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1569 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5635040348160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚765 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚733587 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚347091 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23660068864 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94531385856 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚818917 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23660068864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4766737577472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 289557504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1364 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2607545344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚49 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚947039 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4897360854528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚2 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5635040348160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 130623277056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2643 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚347533 % _timestamp‚1.7884376037361538e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 N loss/SRqqt3/assignment_loss"trainer/global_step2:JPÍçk¹A±b< loss/FBt3/assignment_loss"trainer/global_step2:JPfÖ0ÙA²b< loss/FBt4/assignment_loss"trainer/global_step2:JPƒ‚m§B³b= loss/SRqqt3/detection_loss"trainer/global_step2:JPZ©cMA´b< loss/FRt3/assignment_loss"trainer/global_step2:JPl±EAµb< loss/FRt1/assignment_loss"trainer/global_step2:JPöOJ5C¶b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP–Å…C·b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPÇjÆŽ@¸b; loss/FRt1/detection_loss"trainer/global_step2:JP|X´@¹b; loss/FRt3/detection_loss"trainer/global_step2:JP¢³a¼Cºb> loss/SRqqt2/assignment_loss"trainer/global_step2:JPyÚÎZA»b< loss/FBt2/assignment_loss"trainer/global_step2:JP‚°ã]@¼b; loss/FRt2/detection_loss"trainer/global_step2:JP=4ôå.½) _wandbruntime‚43  _runtime‚43 RÔ\G¾Á 1loss/SRqqt3/assignment_loss‚5.289292736658974 0loss/FBt3/assignment_loss‚2.5724090962182906 0loss/FBt4/assignment_loss‚2.4891624298501522 1loss/SRqqt3/detection_loss‚2.4976306064497855 /loss/FRt3/assignment_loss‚7.031978158389821 /loss/FRt1/assignment_loss‚7.403295959745135 1loss/SRqqt1/assignment_loss‚5.463137600156996 1loss/SRqqt4/assignment_loss‚5.456851132062017 /loss/FRt1/detection_loss‚3.8883053738446463 -loss/FRt3/detection_loss‚2.70413603414565 % _timestamp‚1.7884376037382312e+09 1loss/SRqqt2/assignment_loss‚5.527359173215669 0loss/FBt2/assignment_loss‚2.5456403952378492 .loss/FRt2/detection_loss‚4.576800609762604 1loss/SRqqt2/detection_loss‚2.7603471874548444 -loss/FBt3/detection_loss‚8.13961132331973 .loss/FBt4/detection_loss‚3.029872825686285 &loss/total_loss‚127.15667219234497 epoch‚0 /loss/FRt4/detection_loss‚3.7144597839902747 0loss/SRqqt1/detection_loss‚4.745992860032453 /loss/SRqqt4/detection_loss‚3.18686173035174 _runtime‚ 43.78280683 _step‚3 trainer/global_step‚49 /loss/FBt1/assignment_loss‚2.645445860349215 /loss/FBt1/detection_loss‚13.516886736337955 /loss/FBt2/detection_loss‚13.233555825857016 /loss/FRt2/assignment_loss‚7.336841570364462 /loss/FRt4/assignment_loss‚7.400797182863409ŠC K¿Å 1loss/SRqqt3/detection_loss‚2.4976306064497855 /loss/FRt3/assignment_loss‚7.031978158389821 /loss/FRt1/assignment_loss‚7.403295959745135 1loss/SRqqt1/assignment_loss‚5.463137600156996 1loss/SRqqt4/assignment_loss‚5.456851132062017 /loss/FRt1/detection_loss‚3.8883053738446463 -loss/FRt3/detection_loss‚2.70413603414565 % _timestamp‚1.7884376037382312e+09 1loss/SRqqt2/assignment_loss‚5.527359173215669 0loss/FBt2/assignment_loss‚2.5456403952378492 .loss/FRt2/detection_loss‚4.576800609762604 1loss/SRqqt2/detection_loss‚2.7603471874548444 -loss/FBt3/detection_loss‚8.13961132331973 .loss/FBt4/detection_loss‚3.029872825686285 &loss/total_loss‚127.15667219234497 epoch‚0 /loss/FRt4/detection_loss‚3.7144597839902747 0loss/SRqqt1/detection_loss‚4.745992860032453 /loss/SRqqt4/detection_loss‚3.18686173035174 _runtime‚ 43.78280683 _step‚3 trainer/global_step‚49 /loss/FBt1/assignment_loss‚2.645445860349215 /loss/FBt1/detection_loss‚13.516886736337955 /loss/FBt2/detection_loss‚13.233555825857016 /loss/FRt2/assignment_loss‚7.336841570364462 /loss/FRt4/assignment_loss‚7.400797182863409 1loss/SRqqt3/assignment_loss‚5.289292736658974 0loss/FBt3/assignment_loss‚2.5724090962182906 0loss/FBt4/assignment_loss‚2.4891624298501522)UðAŸÀ‚(:” äÈåÔ‘ƒ÷È gpu.0.gpu‚70.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚44.0 gpu.0.powerWatts‚234.612' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.922399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t÷¿‰ûÁ‚(:ð äÈåÔÛÆéË disk.rbd2.in‚0 disk.sda.out‚ 2.05859375! disk.nvme0c0n1.in‚ 11.73046875 cpu‚24.967249908257884 disk.rbd2.out‚ 2.1953125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚65307% disk./.usageGB‚1100.8064804077148 disk.sda1.in‚0 disk.rbd3.in‚0$ memory_percent‚91.53616428375244# proc.memory.rssMB‚ 2888.77734375 network.recv‚78179 disk.rbd1.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 2.05859375 disk.rbd0.in‚0 disk.dm-0.in‚ 11.73046875 disk.dm-0.out‚ 199.7265625) proc.memory.percent‚17.63169765472412 proc.cpu.threads‚17) disk./.usagePercent‚73.87604653831937 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 11.73046875 disk.nvme0n1.out‚ 199.7265625) proc.memory.availableMB‚ 1386.71484375" disk.nvme0c0n1.out‚ 199.7265625 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 2.05859375ŽéyžÂ‚(:“ óÈåÔ•—É gpu.0.gpu‚80.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚44.0 gpu.0.powerWatts‚96.133' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.226599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×á7Â(:ý óÈåÔâÑ“Ë disk.rbd2.out‚ 3.4658203125 disk.sda3.out‚ 2.74609375) disk./.usagePercent‚73.87604397824403 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚298.9912109375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 2.74609375$ memory_percent‚94.44057941436768( proc.memory.availableMB‚ 910.85546875# proc.memory.rssMB‚ 2888.78515625) proc.memory.percent‚17.63174533843994 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 2.74609375 disk.sda2.out‚0 disk.dm-0.in‚ 11.73046875 cpu‚24.979044452471694 network.sent‚135016 disk.sda2.in‚0# disk.nvme0n1.out‚298.9912109375! disk.nvme0c0n1.in‚ 11.73046875 network.recv‚112910% disk./.usageGB‚1100.8064422607422 disk.rbd1.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚298.9912109375 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 11.73046875 disk.sda3.in‚0w5 ”Ä‚(:‰ ‚ÉåÔ„í•É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚46.0 gpu.0.powerWatts‚232.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.5496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÐ7|Å‚(:ø ‚ÉåÔ®æŒÍ network.recv‚118489) disk./.usagePercent‚73.87587066114335% disk./.usageGB‚1100.8038597106934 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2888.79296875 disk.rbd2.out‚ 4.5400390625 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 11.73046875$ memory_percent‚94.43488121032715 cpu‚24.98604708625157 disk.rbd1.out‚0 disk.sda.out‚ 3.2421875# disk.nvme0n1.out‚398.5244140625 disk.rbd0.in‚0 disk.sda3.out‚ 3.2421875 disk.dm-1.out‚ 3.2421875 disk.rbd2.in‚0 disk.sda1.out‚0 network.sent‚141694 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 11.73046875 disk.sda3.in‚0% disk.nvme0c0n1.out‚398.5244140625' proc.memory.availableMB‚ 911.7890625) proc.memory.percent‚17.63179302215576 disk.dm-0.out‚398.5244140625 proc.cpu.threads‚17 disk.dm-0.in‚ 11.73046875 disk.rbd0.out‚0ùª—¿.Æ) _wandbruntime‚82  _runtime‚82Ì¥Û®ÁÇ» % _timestamp‚1.7884376422036204e+09 _runtime‚ 82.248271125 _step‚4 & lr-AdamW/pg1‚3.173192333434636e-05 & lr-AdamW/pg2‚3.173192333434636e-05 trainer/global_step‚99Ù!' ÅÈ¿ _step‚4 & lr-AdamW/pg1‚3.173192333434636e-05 & lr-AdamW/pg2‚3.173192333434636e-05 trainer/global_step‚99 % _timestamp‚1.7884376422036204e+09 _runtime‚ 82.2482711258.q.É) _wandbruntime‚82  _runtime‚822Êu!Ì#Ê»O P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚921985 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9548547881984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1843831 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1825295 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1600532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚778 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10829528132608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10829528132608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10851828709530 _runtime‚ 82.249183795 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11011258583040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10829554524160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 181730450432 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 trainer/global_step‚99 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1431151 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1431151 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚921988 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300146688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10667801785344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1431157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1843831 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1825295 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 184329892864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚435 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 181428647066 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3256446 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11011258583040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1600967 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9548474681344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240™ø3VÅ'Ë¿O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9548547881984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1843831 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1825295 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1600532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚778 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10829528132608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10829528132608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10851828709530 _runtime‚ 82.249183795 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11011258583040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10829554524160 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 181730450432 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 trainer/global_step‚99 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1431151 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1431151 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚921988 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300146688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10667801785344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1431157 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1843831 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1825295 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 184329892864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚435 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 181428647066 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3256446 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11011258583040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1600967 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9548474681344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚921985 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536‡­,?.Ì) _wandbruntime‚82  _runtime‚82oÇÌ&ÍÆM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11123656464384 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚931184 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300146688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11123656464384 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10957375521382 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11121030617088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚778 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1616804 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1862091 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186162234880 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9643406754816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 257294636544 trainer/global_step‚99 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1862091 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚931181 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10937467837952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚6 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1616395 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚685214 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9643479942144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884376429485795e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1445387 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1843529 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 186162234880 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 183562779136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9386112118272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10774142959616 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚685620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 183562779136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9386140252672 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 257339689472 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 185856267062 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 183257989734 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1445393 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1445387 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968“p}Ð&ÎÊM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1445393 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1445387 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11123656464384 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚931184 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300146688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11123656464384 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10957375521382 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11121030617088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚778 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1616804 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1862091 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186162234880 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9643406754816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 257294636544 trainer/global_step‚99 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1862091 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚931181 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10937467837952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚6 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1616395 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚685214 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9643479942144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884376429485795e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1445387 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1843529 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 186162234880 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 183562779136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9386112118272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10774142959616 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚685620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 183562779136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9386140252672 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 257339689472 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 185856267062 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 183257989734 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16260368147738 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23662166016 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2758883 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2400804 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13913126739968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 306332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4901840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1381 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14294522680320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 381441006592 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 381395940352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16257744442650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884376815247388e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1019016 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14294595880960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16499874699264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16502500533248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16226539791360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2400343 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16226513399808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 275960741888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 273361299456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4883298 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2142957 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2142957 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2142951 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2142951 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15984860498944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1381788 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2758883 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13913154874368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16260368147738 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23662166016 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2758883 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2400804 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13913126739968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 306332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4901840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1381 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14294522680320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 381441006592 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 381395940352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16257744442650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884376815247388e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1019016 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14294595880960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16499874699264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16502500533248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16226539791360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2400343 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16226513399808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 275960741888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 273361299456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4883298 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2142957 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2142957 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2142951 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2142951 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15984860498944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1381788 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2758883 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13913154874368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 275193628160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16609646733312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1381 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14005514793472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2777143 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1390981 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2157193 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 trainer/global_step‚149 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2416311 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16334479496704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1025330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1390984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚10 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16334479496704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 275193628160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14005542927872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 % _timestamp‚1.7884376822727554e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 274738714342 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2416744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16091176245248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16334453105152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14389533681664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4915768 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14389460494336 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16368538664886 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570õÑÙ3Ú&àÔM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 275193628160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16609646733312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1381 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14005514793472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2777143 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1390981 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2157193 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 trainer/global_step‚149 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2416311 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16334479496704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1025330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1390984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚10 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16334479496704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 275193628160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14005542927872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 % _timestamp‚1.7884376822727554e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 274738714342 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2416744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16091176245248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16334453105152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14389533681664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4915768 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14389460494336 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16368538664886 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2758581 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 FÄé¾ % _timestamp‚1.7884377207687716e+09 _runtime‚ 160.813440194 _step‚12 & lr-AdamW/pg1‚6.378437114681743e-05 & lr-AdamW/pg2‚6.378437114681743e-05 trainer/global_step‚199=£uÈê & lr-AdamW/pg1‚6.378437114681743e-05 & lr-AdamW/pg2‚6.378437114681743e-05 trainer/global_step‚199 % _timestamp‚1.7884377207687716e+09 _runtime‚ 160.813440194 _step‚12 {©Œî0ë+ _wandbruntime‚160  _runtime‚160¥™1#Í'ìÇO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3655399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21988490815488 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 367591590912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21301893784576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21623498667008 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21988490815488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3673935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1358088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2630612992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚21991116649472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884377207695377e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3673935 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 364992148480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18532529415168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 364992148480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2854751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400]½· íËO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 364992148480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2854751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3655399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21988490815488 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 367591590912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21301893784576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21623498667008 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21988490815488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3673935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1358088 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2630612992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚21991116649472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884377207695377e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3673935 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 364992148480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18532529415168 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21731438372352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22098262849536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21408234958848 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18624917468672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 366219438950 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21731464763904 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3215592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21408260386816 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3673633 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚14 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 366824477184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1850784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1365253 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22100888696832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6542620 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 369423932928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚199 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2868987 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 510620488704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1364811 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 510573338624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312712192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2868993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2630612992% „Ú&ðÔM KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6542620 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18624917468672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 366219438950 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21731464763904 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3215592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21408260386816 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3673633 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚14 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 366824477184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1850784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1365253 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22100888696832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6542620 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 369423932928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚199 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2868987 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 510620488704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1364811 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 510573338624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312712192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2868993 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2630612992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21731438372352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22098262849536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21408234958848 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634749834240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3999441 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27020483934208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26618977926144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26618952498176 trainer/global_step‚249 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3566557 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23786710059008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 459222439936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 456622997504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23151932090368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1381 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27077447024154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27479732765696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2301388 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27477106931712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23662166016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1698519 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4588987 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27020483934208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4588987 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2630612992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644¹N¬Ñ'ÿËO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2301388 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27477106931712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23662166016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1698519 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4588987 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27020483934208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4588987 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2630612992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27477106931712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23786634761216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2301385 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634702670848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3566551 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3566557 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27020510325760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 458469098010 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312712192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27074823319066 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 _runtime‚ 200.081276138 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27020510325760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634749834240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3999441 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27020483934208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26618977926144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26618952498176 trainer/global_step‚249 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3566557 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23786710059008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 459222439936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 456622997504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23151932090368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1381 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27077447024154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27479732765696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0zˆº0€+ _wandbruntime‚200  _runtime‚200ô#!Ë×&ÑM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4607247 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26725293672448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 457700163558 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27128450031104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27128423639552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1381 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23244320143872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27128423639552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 458455326208 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3580787 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2310584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2630612992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 _step‚18 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3580787 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8169472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 % _timestamp‚1.7884377607857006e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3580793 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 461054781952 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8188040 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4588685 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637232406016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4588685 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2310581 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1705217 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23881552549888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23244348278272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4015801 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4015334 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 637279556096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8169472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚782 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27185617541302 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26725319100416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27128450031104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 _step‚18 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3580787 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8169472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 % _timestamp‚1.7884377607857006e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3580793 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 461054781952 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8188040 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4588685 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637232406016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4588685 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2310581 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1705217 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23881552549888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23244348278272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4015801 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4015334 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 637279556096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8169472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚782 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27185617541302 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26725319100416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27128450031104 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 549949822618 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32417495592960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2036720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312712192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31936036639744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31936011211776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4278351 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2630612992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2761188 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4278351 % _timestamp‚1.7884377995584135e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32965723047936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4797458 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32485986462362 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32417469201408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27771334765568 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 761346400768 trainer/global_step‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32417469201408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32965723047936 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5485503 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 761391467008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 548253846528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27771362899968 _runtime‚ 239.603403228 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1381 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28532754366976 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 547351545498 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23662166016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28532681166336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32968348881920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32417495592960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32968348881920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5504039 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 548253846528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 550853288960 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9782396 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 550853288960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968?bg4gËO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4278351 % _timestamp‚1.7884377995584135e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32965723047936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4797458 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32485986462362 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32417469201408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27771334765568 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 761346400768 trainer/global_step‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32417469201408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32965723047936 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5485503 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 761391467008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 548253846528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27771362899968 _runtime‚ 239.603403228 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1381 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28532754366976 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 547351545498 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23662166016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28532681166336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32968348881920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32417495592960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32968348881920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5504039 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 548253846528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 550853288960 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9782žÍ3tj396 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 550853288960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4278357 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5485503 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5504039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2036273 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚782 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 549949822618 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32417495592960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2036720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312712192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31936036639744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31936011211776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4278351 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2630612992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2761188 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 „Y`”‚(:‰ §ÊåÔ™ùÉ gpu.0.gpu‚68.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚45.0 gpu.0.powerWatts‚230.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.1152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó³ãH‘‚(:ü §ÊåÔŽäŽË$ disk./.usageGB‚1100.808853149414 disk.sda3.out‚ 10.4765625 disk.drbd1777.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.nvme0n1.in‚ 11.734375' disk.nvme0c0n1.out‚1198.29248046875( proc.memory.availableMB‚ 910.05078125 disk.rbd2.out‚13.90771484375 disk.sda.out‚ 10.4765625 disk.rbd3.out‚0 disk.sda3.in‚0$ memory_percent‚94.44549083709717 cpu‚25.02503175208353 network.sent‚386957 network.recv‚177249 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.632007598876953) disk./.usagePercent‚73.87620577500567 disk.sda.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚1198.29248046875 disk.dm-1.out‚ 10.4765625 disk.rbd1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 11.734375 disk.dm-0.in‚ 11.734375" disk.dm-0.out‚1198.29248046875! proc.memory.rssMB‚ 2888.828125 disk.dm-1.in‚0 disk.sda1.in‚0øNc0’+ _wandbruntime‚240  _runtime‚240ŠÃ×&“ÑM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33078120929280 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32525408906752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2770381 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚22 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5503737 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 551779165494 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 _runtime‚ 240.352818035 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33078120929280 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4813338 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28627671094272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 550086175232 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28627597906944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2630612992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5522299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 552685630976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2043382 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27863750953472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32594156979510 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚782 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33075495081984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27863722819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312712192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4292587 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5503737 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2770384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32042352386048 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9814892 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32525435298304Á%”ÖÛ&”ÕM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32042352386048 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9814892 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32525435298304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33078120929280 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32525408906752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2770381 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚22 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5503737 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 551779165494 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 _runtime‚ 240.352818035 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33078120929280 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4813338 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28627671094272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 550086175232 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28627597906944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2630612992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5522299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 552685630976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2043382 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27863750953472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32594156979510 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚782 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33075495081984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27863722819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312712192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4292587 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5503737 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2770384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296‚/P¦0•+ _wandbruntime‚240  _runtime‚240Sã˜ÑN–È trainer/global_step‚299 /loss/FRt1/detection_loss‚2.5016196656321723 .loss/FRt2/assignment_loss‚7.38522883521186 /loss/FRt3/assignment_loss‚7.058606918041523 /loss/FRt4/assignment_loss‚7.161296451300905 /loss/FRt2/detection_loss‚3.5615815785196094 /loss/FBt1/detection_loss‚3.6063527442514896 0loss/SRqqt1/assignment_loss‚2.84237545132637 2loss/SRqqt4/assignment_loss‚2.4221373945474625 .loss/FRt3/detection_loss‚4.011465622828557 /loss/FRt4/detection_loss‚2.4818281377093836 _runtime‚ 240.354087717 1loss/SRqqt2/assignment_loss‚2.640523267955315 1loss/SRqqt3/assignment_loss‚3.282631507733973 .loss/FBt3/detection_loss‚5.100426728098557 0loss/FBt4/assignment_loss‚0.6431160213348145 1loss/FBt1/assignment_loss‚0.13507141778754886 0loss/FBt3/assignment_loss‚0.4639099869638248 0loss/SRqqt3/detection_loss‚3.985060873554974 %loss/total_loss‚89.11716484392838 $ _timestamp‚1.788437800309522e+09 1loss/FBt2/assignment_loss‚0.41956309212319964 0loss/SRqqt1/detection_loss‚4.115896948426962 .loss/FBt4/detection_loss‚4.699387716284645 epoch‚0 0loss/FRt1/assignment_loss‚7.3353016338651145 _step‚23 1loss/SRqqt2/detection_loss‚3.9311378176619365 1loss/SRqqt4/detection_loss‚0.7008522970458476 -loss/FBt2/detection_loss‚8.63179273572233q$ÐÑR—Ì 1loss/SRqqt2/detection_loss‚3.9311378176619365 1loss/SRqqt4/detection_loss‚0.7008522970458476 -loss/FBt2/detection_loss‚8.63179273572233 trainer/global_step‚299 /loss/FRt1/detection_loss‚2.5016196656321723 .loss/FRt2/assignment_loss‚7.38522883521186 /loss/FRt3/assignment_loss‚7.058606918041523 /loss/FRt4/assignment_loss‚7.161296451300905 /loss/FRt2/detection_loss‚3.5615815785196094 /loss/FBt1/detection_loss‚3.6063527442514896 0loss/SRqqt1/assignment_loss‚2.84237545132637 2loss/SRqqt4/assignment_loss‚2.4221373945474625 .loss/FRt3/detection_loss‚4.011465622828557 /loss/FRt4/detection_loss‚2.4818281377093836 _runtime‚ 240.354087717 1loss/SRqqt2/assignment_loss‚2.640523267955315 1loss/SRqqt3/assignment_loss‚3.282631507733973 .loss/FBt3/detection_loss‚5.100426728098557 0loss/FBt4/assignment_loss‚0.6431160213348145 1loss/FBt1/assignment_loss‚0.13507141778754886 0loss/FBt3/assignment_loss‚0.4639099869638248 0loss/SRqqt3/detection_loss‚3.985060873554974 %loss/total_loss‚89.11716484392838 $ _timestamp‚1.788437800309522e+09 1loss/FBt2/assignment_loss‚0.41956309212319964 0loss/SRqqt1/detection_loss‚4.115896948426962 .loss/FBt4/detection_loss‚4.699387716284645 epoch‚0 0loss/FRt1/assignment_loss‚7.3353016338651145 _step‚23 }Œ”˜‚(:‰ ¶ÊåÔÒ›ƒÉ gpu.0.gpu‚69.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚47.0 gpu.0.powerWatts‚233.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.7654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ÿ€ÿ™‚(:ô ¶ÊåÔÔàÃÌ disk.sda2.in‚0 disk.nvme0c0n1.in‚ 11.734375 disk.dm-1.out‚11.0625$ memory_percent‚94.44451332092285 proc.cpu.threads‚17 disk.rbd3.in‚0 network.recv‚182879% disk./.usageGB‚1100.8089065551758 disk.sda.in‚0 disk.sda3.in‚0" disk.dm-0.out‚1297.54833984375' proc.memory.availableMB‚ 910.2109375 cpu‚25.026100497685373 disk.rbd3.out‚0 disk.nvme0n1.in‚ 11.734375 disk.rbd0.in‚0 disk.dm-0.in‚ 11.734375 disk.rbd1.in‚0 network.sent‚425336) disk./.usagePercent‚73.87620935911114 disk.drbd1777.in‚0 disk.sda.out‚11.0625 disk.sda1.in‚0% disk.nvme0n1.out‚1297.54833984375 disk.rbd1.out‚0! proc.memory.rssMB‚ 2888.828125 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.632007598876953 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚14.98193359375 disk.sda3.out‚11.0625' disk.nvme0c0n1.out‚1297.54833984375ýÅ”š‚(:‰ ÅÊåÔ¾Þ›É gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mö¼‚ ›‚(:‚ ÅÊåÔ¢õ×Ë proc.cpu.threads‚17% disk.nvme0n1.out‚1396.71826171875 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚1396.71826171875) disk./.usagePercent‚73.87620961511868 disk.sda2.out‚0 disk.nvme0n1.in‚ 11.734375$ memory_percent‚94.44048404693604# proc.memory.rssMB‚ 2888.83203125 disk.sda.out‚ 11.79296875 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1396.71826171875 disk.dm-0.in‚ 11.734375 disk.drbd1777.in‚0 disk.rbd1.out‚0 network.sent‚464476 disk.dm-1.in‚0* proc.memory.percent‚17.632031440734863 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 11.734375 cpu‚25.028295901009912 network.recv‚187995$ disk./.usageGB‚1100.808910369873 disk.dm-1.out‚ 11.79296875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 11.79296875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚15.07177734375( proc.memory.availableMB‚ 910.87109375óøý±0œ+ _wandbruntime‚278  _runtime‚278îýÆÀ ' lr-AdamW/pg1‚0.00011186304286552405 ' lr-AdamW/pg2‚0.00011186304286552405 trainer/global_step‚349 % _timestamp‚1.7884378388172214e+09 _runtime‚ 278.861922678 _step‚24Ý9¡ÊžÄ ' lr-AdamW/pg1‚0.00011186304286552405 ' lr-AdamW/pg2‚0.00011186304286552405 trainer/global_step‚349 % _timestamp‚1.7884378388172214e+09 _runtime‚ 278.861922678 _step‚24\åv 0Ÿ+ _wandbruntime‚278  _runtime‚278Øb秠ËO _runtime‚ 278.862817825 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37814480860160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 887790882816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 642484137984 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 639884695552 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5594715 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚349 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6400555 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 638832270106 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_ba=ëú*"tch_start/requested_bytes.all.current‚ 2623705088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38454339164160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32390737440768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37814454468608 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4990157 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37891902195482 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11409248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚783 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37814480860160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4990157 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6400555 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37814454468608 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4990157 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37891902195482 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11409248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚783 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37814480860160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4990157 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6400555 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38454339164160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32390737440768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644·Ù;#0¢+ _wandbruntime‚279  _runtime‚279OÖíµÛ&£ÕM \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 890329634304 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2380839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32483153628672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5004387 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6418789 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38002696417718 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6418789 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38566737045504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 _step‚26 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5004387 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37922394173952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5004393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 644316480000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5004393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 641717024256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37359411099648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6437351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5611023 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2380423 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 279.607403094 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 890374687232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884378395622518e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37922420565504 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 644316480000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3230181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38564111198208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37922420565504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5610604 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37359436527616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 640661612774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 643259890102 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37922394173952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016›¶ñôÓ¤ÙM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33373455128576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38566737045504 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5004387 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37922394173952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5004393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 644316480000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5004393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 641717024256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37359411099648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6437351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5611023 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2380423 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 279.607403094 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 890374687232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884378395622518e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37922420565504 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.alloc Ú ated‚ 644316480000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3230181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38564111198208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37922420565504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5610604 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37359436527616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 640661612774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 643259890102 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37922394173952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 890329634304 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2380839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32483153628672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5004387 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6418789 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38002696417718 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6418789 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38566737045504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 _step‚26 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5701957 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43211439735808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 % _timestamp‚1.7884378782731907e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 _step‚29 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 _runtime‚ 318.318159287 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5701951 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43942955280384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3680788 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7334143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6394127 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 trainer/global_step‚399 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 732911271834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43942955280384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3680788 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7334143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6394127 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 trainer/global_step‚399 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 732911271834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5701957 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43211439735808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 % _timestamp‚1.7884378782731907e+09Gžâ0´+ _wandbruntime‚319  _runtime‚319/]…/Ü&µÖM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 735947329024 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5716193 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37102528169472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44055353161728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6409606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43319405832704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 733347873280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚783 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5716187 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43319379441152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44052727314432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3689981 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 733347873280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37102556303872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42676495241216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 _step‚30 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1017198551552 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5716193 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7352403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚408 $ _timestamp‚1.788437879022969e+09 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38119754855424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7333841 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44055353161728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚411 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2720033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43411235855926 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43408612150630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7333841 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38119681668096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42676469813248 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 trainer/global_step‚399 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5716187 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7352403 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1017153498624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6410017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3689981 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 733347873280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37102556303872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42676495241216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 _step‚30 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1017198551552 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5716193 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7352403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚408 $ _timestamp‚1.788437879022969e+09 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38119754855424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7333841 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44055353161728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚411 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2720033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43411235855926 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43408612150630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7333841 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38119681668096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42676469813248 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 trainer/global_step‚399 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5716187 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7352403 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1017153498624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6410017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44055353161728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6409606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43319405832704 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 733347873280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚783 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5716187 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43319379441152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644胓Y0·+ _wandbruntime‚319  _runtime‚319"ÖªM¸Ç 1loss/FBt1/assignment_loss‚0.33834123893358414 /loss/FRt4/detection_loss‚1.9304452368723495 .loss/FBt3/detection_loss‚4.565795463509858 %loss/total_loss‚86.87739344188543 1loss/SRqqt3/assignment_loss‚2.766293886729649 0loss/FBt2/assignment_loss‚0.5944866329615858 .loss/FRt1/detection_loss‚3.225610127362112 .loss/FRt2/detection_loss‚2.864770288175007 /loss/FRt2/assignment_loss‚7.336850733127234 1loss/FBt3/assignment_loss‚0.24582969553338785 0loss/FBt4/assignment_loss‚0.4701333255249595 1loss/SRqqt4/detection_loss‚1.3034971947574212 /loss/FRt4/assignment_loss‚7.265569468906947 0loss/SRqqt3/detection_loss‚4.355541349308831 .loss/FBt1/detection_loss‚2.392749230350918 epoch‚0 % _timestamp‚1.7884378790251405e+09 1loss/SRqqt1/assignment_loss‚2.899748921394348 0loss/SRqqt1/detection_loss‚4.444983359745571 _runtime‚ 319.069710911 _step‚31 /loss/FRt3/detection_loss‚3.4016243144869804 .loss/FRt1/assignment_loss‚7.27595258752505 1loss/SRqqt2/assignment_loss‚2.457739756063179 2loss/SRqqt4/assignment_loss‚2.8019916221692966 .loss/FBt2/detection_loss‚11.98869227617979 /loss/FRt3/assignment_loss‚7.169725179672241 0loss/SRqqt2/detection_loss‚2.725157739456605 .loss/FBt4/detection_loss‚2.055863813138515 trainer/global_step‚399¿rQ¹Ë 0loss/SRqqt3/detection_loss‚4.355541349308831 .loss/FBt1/detection_loss‚2.392749230350918 epoch‚0 % _timestamp‚1.7884378790251405e+09 1loss/SRqqt1/assignment_loss‚2.899748921394348 0loss/SRqqt1/detection_loss‚4.444983359745571 _runtime‚ 319.069710911 _step‚31 /loss/FRt3/detection_loss‚3.4016243144869804 .loss/FRt1/assignment_loss‚7.27595258752505 1loss/SRqqt2/assignment_loss‚2.457739756063179 2loss/SRqqt4/assignment_loss‚2.8019916221692966 .loss/FBt2/detection_loss‚11.98869227617979 /loss/FRt3/assignment_loss‚7.169725179672241 0loss/SRqqt2/detection_loss‚2.725157739456605 .loss/FBt4/detection_loss‚2.055863813138515 trainer/global_step‚399 1loss/FBt1/assignment_loss‚0.33834123893358414 /loss/FRt4/detection_loss‚1.9304452368723495 .loss/FBt3/detection_loss‚4.565795463509858 %loss/total_loss‚86.87739344188543 1loss/SRqqt3/assignment_loss‚2.766293886729649 0loss/FBt2/assignment_loss‚0.5944866329615858 .loss/FRt1/detection_loss‚3.225610127362112 .loss/FRt2/detection_loss‚2.864770288175007 /loss/FRt2/assignment_loss‚7.336850733127234 1loss/FBt3/assignment_loss‚0.24582969553338785 0loss/FBt4/assignment_loss‚0.4701333255249595 1loss/SRqqt4/detection_loss‚1.3034971947574212 /loss/FRt4/assignment_loss‚7.265569468906947—Ú©6”º‚(:‰ ËåÔâãåÉ gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚47.0 gpu.0.powerWatts‚276.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W¹æ(û»‚(:ð ËåÔ¹ÓÂÌ disk.rbd3.in‚0' proc.memory.availableMB‚ 910.4296875# proc.memory.rssMB‚ 2888.83984375 cpu‚25.029720897336404! disk.nvme0n1.out‚ 1695.0078125 disk.nvme0c0n1.in‚ 11.734375 disk.rbd1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.87620859108854 disk.rbd2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 1695.0078125 disk.drbd1777.in‚0 disk.rbd1.in‚0$ memory_percent‚94.44317817687988* proc.memory.percent‚17.632079124450684 network.recv‚207483 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 1695.0078125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 18.654296875 disk.sda1.in‚0$ disk./.usageGB‚1100.808895111084 disk.sda3.out‚ 14.1484375 disk.dm-0.in‚ 11.734375 disk.dm-1.out‚ 14.1484375 disk.drbd1777.out‚0 disk.sda.out‚ 14.1484375 network.sent‚533066 disk.sda3.in‚0 disk.nvme0n1.in‚ 11.734375 disk.sda.in‚0C¡œ×Ÿ¼‚(:” ËåÔ¿ŸÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.466' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.293200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!c´í½‚(:ø ËåÔž¢…Ì disk.sda1.out‚0& proc.memory.availableMB‚ 910.171875 cpu‚25.02993543030016 network.sent‚583042 network.recv‚213658 disk.sda3.out‚ 14.51171875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0% disk.nvme0c0n1.out‚1695.392578125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2888.83984375 disk.dm-0.in‚ 11.734375 disk.dm-1.out‚ 14.51171875 disk.drbd1777.in‚0 disk.rbd2.in‚0$ memory_percent‚94.44475173950195 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 11.734375 disk.dm-0.out‚1695.392578125 disk.rbd1.out‚0 disk.nvme0n1.in‚ 11.734375 disk.sda2.in‚0* proc.memory.percent‚17.632079124450684) disk./.usagePercent‚73.87620884709608% disk./.usageGB‚1100.8088989257812 disk.rbd2.out‚ 19.767578125 disk.rbd3.in‚0# disk.nvme0n1.out‚1695.392578125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 14.51171875ëÊ„”0¾+ _wandbruntime‚357  _runtime‚357ãAÒÿ½ & lr-AdamW/pg1‚0.0001439154906779951 & lr-AdamW/pg2‚0.0001439154906779951 trainer/global_step‚449 $ _timestamp‚1.788437917503023e+09 _runtime‚ 357.547743163 _step‚32¦,ZÇÀÁ & lr-AdamW/pg1‚0.0001439154906779951 & lr-AdamW/pg2‚0.0001439154906779951 trainer/global_step‚449 $ _timestamp‚1.788437917503023e+09 _runtime‚ 357.547743163 _step‚32 PôHK0Á+ _wandbruntime‚357  _runtime‚357ij܈ðÂÌO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚71103488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6413757 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49431571396608 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 823146393600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48608425003008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49434197230592 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6413751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3049979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14644410 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48608425003008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49434197230592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8249195 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚783 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48708981071898 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49431571396608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48608451394560 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8230659 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41629570925568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8249195 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7190996 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 321013760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788437917503841e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6413751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48608451394560 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 H?DeviceStatsMonitor.on_train_batch_startmàt â/inactive_split.all.peak‚1382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14662952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚449 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1141425120256 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 823146393600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42770967911424 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2639001600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41629542791168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 357.548851841 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 824391996442 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47887212780544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7190564 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 821793719322ÈcîÖ'ÃÐO H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48608425003008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49434197230592 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6413751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3049979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14644410 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48608425003008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3292528640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49434197230592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8249195 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚783 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48708981071898 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49431571396608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48608451394560 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8230659 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41629570925568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8249195 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7190996 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 321013760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788437917503841e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6413751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48608451394560 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14662952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚449 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1141425120256 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 823146393600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42770967911424 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2639001600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41629542791168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 357.548851841 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 824391996442 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47887212780544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2644 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7190564 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 821793719322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚71103488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6413757 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49431571396608 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 823146393600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020!¿_„0Ä+ _wandbruntime‚358  _runtime‚358ºKÝ&Å×M NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7206829 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48716391099904 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8267455 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42865973046272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48817151588838 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1143969014272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48819775294134 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47993553954816 % _timestamp‚1.7884379182570758e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 824978722304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48716364708352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 _runtime‚ 358.302412188 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2639001600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 827578178048 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8248893 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1144014067200 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14676880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6427987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚783 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 827578178048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41721958979072 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49543969277952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42865899858944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8248893 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 823623061990 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48716364708352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7206416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 trainer/global_step‚449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49543969277952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3057045 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚413 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41721930844672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 824978722304 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚34 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49541343430656 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6427987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47993528526848 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 826221339318 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016¢¸ÄÆÛM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3057045 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚45052928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚413 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41721930844672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 824978722304 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚34 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49541343430656 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6427987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47993528526848 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 826221339318 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7206829 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48716391099904 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8267455 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42865973046272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48817151588838 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1143969014272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48819775294134 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47993553954816 % _timestamp‚1.7884379182570758e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 824978722304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48716364708352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3292528640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 _runtime‚ 358.302412188 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2639001600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 827578178048 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8248893 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1144014067200 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14676880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6427987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 Msx—É CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚783 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 827578178048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41721958979072 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49543969277952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42865899858944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8248893 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 823623061990 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48716364708352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7206416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 trainer/global_step‚449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49543969277952"B40Ç+ _wandbruntime‚358  _runtime‚358ö´mOÈÉ 1loss/SRqqt4/detection_loss‚1.3415425177732392 $loss/total_loss‚85.3751977089127 1loss/SRqqt4/assignment_loss‚2.719822436571121 0loss/FBt4/assignment_loss‚0.7577158730153156 /loss/FRt2/detection_loss‚3.4300930042599522 2loss/SRqqt1/assignment_loss‚2.2014860601136177 2loss/SRqqt2/assignment_loss‚2.4644616448879244 -loss/FRt1/detection_loss‚2.56086702644825 0loss/SRqqt1/detection_loss‚4.456445586500746 % _timestamp‚1.7884379182590625e+09 /loss/FRt2/assignment_loss‚7.309324430864911 /loss/FRt3/assignment_loss‚6.949798757379705 /loss/FRt3/detection_loss‚3.0272799621928823 /loss/FRt4/detection_loss‚2.4399129079900823 1loss/SRqqt2/detection_loss‚2.7323376300558446 .loss/FRt1/assignment_loss‚6.70222840874882 1loss/SRqqt3/assignment_loss‚2.974756653492267 /loss/FBt2/assignment_loss‚0.688571662812527 /loss/FBt4/detection_loss‚0.7083499583682055 epoch‚0 trainer/global_step‚449 /loss/FRt4/assignment_loss‚7.225408428120163 /loss/FBt1/detection_loss‚2.2713793236762285 -loss/FBt2/detection_loss‚9.45527418596404 _step‚35 1loss/FBt3/assignment_loss‚0.35076963225838703 /loss/FBt3/detection_loss‚6.8130874037742615 _runtime‚ 358.303646142 1loss/FBt1/assignment_loss‚0.30478031649078563 0loss/SRqqt3/detection_loss‚5.489503897153414&º`rSÉÍ 1loss/FBt1/assignment_loss‚0.30478031649078563 0loss/SRqqt3/detection_loss‚5.489503897153414 1loss/SRqqt4/detection_loss‚1.3415425177732392 $loss/total_loss‚85.3751977089127 1loss/SRqqt4/assignment_loss‚2.719822436571121 0loss/FBt4/assignment_loss‚0.7577158730153156 /loss/FRt2/detection_loss‚3.4300930042599522 2loss/SRqqt1/assignment_loss‚2.2014860601136177 2loss/SRqqt2/assignment_loss‚2.4644616448879244 -loss/FRt1/detection_loss‚2.56086702644825 0loss/SRqqt1/detection_loss‚4.456445586500746 % _timestamp‚1.7884379182590625e+09 /loss/FRt2/assignment_loss‚7.309324430864911 /loss/FRt3/assignment_loss‚6.949798757379705 /loss/FRt3/detection_loss‚3.0272799621928823 /loss/FRt4/detection_loss‚2.4399129079900823 1loss/SRqqt2/detection_loss‚2.7323376300558446 .loss/FRt1/assignment_loss‚6.70222840874882 1loss/SRqqt3/assignment_loss‚2.974756653492267 /loss/FBt2/assignment_loss‚0.688571662812527 /loss/FBt4/detection_loss‚0.7083499583682055 epoch‚0 trainer/global_step‚449 /loss/FRt4/assignment_loss‚7.225408428120163 /loss/FBt1/detection_loss‚2.2713793236762285 -loss/FBt2/detection_loss‚9.45527418596404 _step‚35 1loss/FBt3/assignment_loss‚0.35076963225838703 /loss/FBt3/detection_loss‚6.8130874037742615 _runtime‚ 358.303646142#ÊâBƒ“Ê‚(:ˆ ŸËåÔèƒÉ gpu.0.gpu‚56.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚46.0 gpu.0.powerWatts‚94.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Ü Ë‚(:ÿ ŸËåÔÜ€£Ë network.sent‚589597 network.recv‚218169% disk./.usageGB‚1100.8090209960938 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚94.44856643676758 disk.rbd2.out‚ 20.998046875 disk.sda.in‚0 disk.nvme0n1.in‚ 11.734375' disk.nvme0c0n1.out‚1794.60986328125 disk.drbd1777.in‚0 disk.rbd1.out‚0% disk.nvme0n1.out‚1794.60986328125 disk.dm-1.in‚0& proc.memory.availableMB‚ 909.546875 cpu‚25.030169840503756 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2888.83984375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 11.734375 disk.dm-1.out‚ 15.12109375) disk./.usagePercent‚73.87621703933718 disk.sda.out‚ 15.12109375 disk.sda1.out‚0 disk.sda3.out‚ 15.12109375 disk.dm-0.in‚ 11.734375 disk.rbd3.out‚0 disk.sda3.in‚0" disk.dm-0.out‚1794.60986328125* proc.memory.percent‚17.632079124450684-+§`žÌ‚(:“ ®ËåÔâÈÓÈ gpu.0.gpu‚81.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚46.0 gpu.0.powerWatts‚257.821' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.56420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èB-ýÍ‚(:ò ®ËåÔ¤àçÌ disk.sda3.in‚0# disk.nvme0c0n1.out‚ 1894.0703125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 15.82421875$ memory_percent‚94.44723129272461 disk.sda3.out‚ 15.82421875 disk.rbd0.in‚0 cpu‚25.031334280048355 disk.dm-0.in‚ 11.734375 disk.rbd1.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 11.734375* proc.memory.percent‚17.632079124450684 disk.dm-0.out‚ 1894.0703125 disk.sda.out‚ 15.82421875 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚639038 network.recv‚223437 disk.rbd2.out‚ 21.029296875& proc.memory.availableMB‚ 909.765625( disk./.usagePercent‚73.8762157592995 disk.drbd1777.in‚0! disk.nvme0n1.out‚ 1894.0703125# proc.memory.rssMB‚ 2888.83984375% disk./.usageGB‚1100.8090019226074 disk.sda2.in‚0 disk.nvme0n1.in‚ 11.734375ÁªX”΂(:‰ ½ËåÔ–ùœÉ gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚47.0 gpu.0.powerWatts‚268.334' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_`ÇûÏ‚(:ð ½ËåÔ¹ˆ™Ë disk.dm-1.in‚0 disk.dm-1.out‚ 16.62890625 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚645421 disk.dm-0.out‚ 1993.328125 disk.rbd2.in‚0 disk.rbd2.out‚ 22.244140625 disk.nvme0n1.out‚ 1993.328125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚227772% disk./.usageGB‚1100.8090019226074 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 1993.328125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 910.03515625 cpu‚25.03160654949774( disk./.usagePercent‚73.8762157592995 disk.nvme0n1.in‚ 11.734375# proc.memory.rssMB‚ 2888.83984375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375 disk.sda.in‚0 disk.sda.out‚ 16.62890625$ memory_percent‚94.44558620452881 proc.cpu.threads‚17 disk.sda3.out‚ 16.62890625 disk.nvme0c0n1.in‚ 11.734375* proc.memory.percent‚17.632079124450684 ”rÜ0Ð+ _wandbruntime‚396  _runtime‚396ŠxžÆÑÀ % _timestamp‚1.7884379568741267e+09 _runtime‚ 396.918821401 _step‚36 ' lr-AdamW/pg1‚0.00015994171458423066 ' lr-AdamW/pg2‚0.00015994171458423066 trainer/global_step‚499\ vtÊÒÄ ' lr-AdamW/pg1‚0.00015994171458423066 ' lr-AdamW/pg2‚0.00015994171458423066 trainer/global_step‚499 % _timestamp‚1.7884379568741267e+09 _runtime‚ 396.918821401 _step‚36$¦‰ùg0Ó+ _wandbruntime‚396  _runtime‚396æNdßÓ'ÔÍO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7988116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9164247 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54922813346816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54920187512832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3387286 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 914777242624 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 % _timestamp‚1.7884379568749022e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54005410270208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4600385 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚71103488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54117520510106 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9145711 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54922813346816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46248973600768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1267986833920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9145711 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7125551 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1385 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53204246066176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9164247 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47517003403776 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47516932300288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2639001600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54120144215194 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 913274443930ˆGþ×'ÕÑO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 914777242624 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 % _timestamp‚1.7884379568749022e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54005410270208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4600385 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚71103488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚54117520510106 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9145711 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54922813346816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46248973600768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1267986833920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9145711 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7125551 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1385 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53204246066176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9164247 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47517003403776 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47516932300288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2639001600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54120144215194 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 913274443930 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7988116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9164247 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54922813346816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54920187512832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3387286%ØðÐ0Ö+ _wandbruntime‚397  _runtime‚397Þ…–V×ÖM QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9182507 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4609581 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54113376367104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚499 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46341361654272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 917702063926 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚$][†0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54228314732342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9182507 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54113349975552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8003917 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7139793 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54225691027046 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9163945 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚786 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 919209027072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47611927942656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55032585394176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53310587240448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55032585394176 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 919209027072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884379576142545e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8003507 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46341333519872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 _step‚38 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 916609571328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54113349975552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1270566288384 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296ˆÌ´à&ØÚM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54113349975552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1270566288384 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9182507 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4609581 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54113376367104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚499 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23662166016 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46341361654272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 917702063926 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54228314732342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2644 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1570 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9182507 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54113349975552 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8003917 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7139793 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54225691027046 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9163945 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚786 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 919209027072 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47611927942656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55032585394176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3292528640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53310587240448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55032585394176 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 919209027072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2644 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884379576142545e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8003507 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46341333519872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23662166016 _step‚38 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 916609571328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2&xÄè0Ù+ _wandbruntime‚397  _runtime‚397ÇK©&KÚÅ .loss/FRt4/detection_loss‚2.522872828368871 /loss/FRt2/detection_loss‚3.1162811464733546 .loss/FBt3/detection_loss‚4.186711298767477 0loss/SRqqt3/detection_loss‚4.541325888037681 trainer/global_step‚499 /loss/FRt1/assignment_loss‚6.441111239981144 1loss/SRqqt1/assignment_loss‚2.788203959195119 1loss/SRqqt1/detection_loss‚2.7813092335215153 _step‚39 2loss/SRqqt2/assignment_loss‚2.7639737725257874 1loss/FBt3/assignment_loss‚0.26201371633069215 0loss/SRqqt2/detection_loss‚2.661393141660553 1loss/SRqqt4/detection_loss‚1.8222597329080559 2loss/SRqqt4/assignment_loss‚2.6996910447504985 0loss/FBt2/assignment_loss‚0.6671800867400386 /loss/FRt1/detection_loss‚3.0553415880558337 epoch‚0 .loss/FRt2/assignment_loss‚6.68822225994534 /loss/FRt4/assignment_loss‚7.118343236311427 -loss/FRt3/detection_loss‚3.43110367986891 .loss/FBt1/detection_loss‚2.252934714997722 $ _timestamp‚1.788437957616487e+09 /loss/FRt3/assignment_loss‚7.005849944220649 /loss/FBt2/detection_loss‚10.684881781989878 $loss/total_loss‚84.8129523793592 _runtime‚ 397.661033128 ,loss/FBt4/detection_loss‚3.4289194020358 1loss/SRqqt3/assignment_loss‚2.962217962741852 1loss/FBt1/assignment_loss‚0.28825372477757827 0loss/FBt4/assignment_loss‚0.6425569951534271C÷qµOÛÉ /loss/FRt3/assignment_loss‚7.005849944220649 /loss/FBt2/detection_loss‚10.684881781989878 $loss/total_loss‚84.8129523793592 _runtime‚ 397.661033128 ,loss/FBt4/detection_loss‚3.4289194020358 1loss/SRqqt3/assignment_loss‚2.962217962741852 1loss/FBt1/assignment_loss‚0.28825372477757827 0loss/FBt4/assignment_loss‚0.6425569951534271 .loss/FRt4/detection_loss‚2.522872828368871 /loss/FRt2/detection_loss‚3.1162811464733546 .loss/FBt3/detection_loss‚4.186711298767477 0loss/SRqqt3/detection_loss‚4.541325888037681 trainer/global_step‚499 /loss/FRt1/assignment_loss‚6.441111239981144 1loss/SRqqt1/assignment_loss‚2.788203959195119 1loss/SRqqt1/detection_loss‚2.7813092335215153 _step‚39 2loss/SRqqt2/assignment_loss‚2.7639737725257874 1loss/FBt3/assignment_loss‚0.26201371633069215 0loss/SRqqt2/detection_loss‚2.661393141660553 1loss/SRqqt4/detection_loss‚1.8222597329080559 2loss/SRqqt4/assignment_loss‚2.6996910447504985 0loss/FBt2/assignment_loss‚0.6671800867400386 /loss/FRt1/detection_loss‚3.0553415880558337 epoch‚0 .loss/FRt2/assignment_loss‚6.68822225994534 /loss/FRt4/assignment_loss‚7.118343236311427 -loss/FRt3/detection_loss‚3.43110367986891 .loss/FBt1/detection_loss‚2.252934714997722 $ _timestamp‚1.788437957616487e+09'e]¨C”Ü‚(:‰ ÌËåÔÔÍÉ gpu.0.gpu‚74.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KR/S Ý‚(:€ ÌËåÔñ¥¢Ì disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375 disk.dm-1.out‚ 17.02734375 disk.drbd1777.in‚0$ memory_percent‚94.44699287414551 network.recv‚231904 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.632079124450684 cpu‚25.032695787893083 disk.sda.out‚ 17.02734375 disk.sda2.out‚0 disk.sda3.out‚ 17.02734375 disk.drbd1777.out‚0 disk.rbd1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 11.734375' disk.nvme0c0n1.out‚1993.99072265625 disk.rbd2.out‚ 23.287109375' proc.memory.availableMB‚ 909.8046875 network.sent‚650627% disk.nvme0n1.out‚1993.99072265625 disk.rbd0.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2888.83984375 disk.nvme0n1.in‚ 11.734375) disk./.usagePercent‚73.87607137105019 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1100.8068504333496 disk.sda.in‚0" disk.dm-0.out‚1993.99072265625 proc.cpu.threads‚17ìÆLŸÞ‚(:” ÛËåÔˆ¨›É gpu.0.gpu‚88.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚46.0 gpu.0.powerWatts‚141.452' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.290399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¦" ß‚(:ÿ ÛËåÔû›Ë disk.sda2.out‚0' disk.nvme0c0n1.out‚2093.14697265625 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.rbd2.out‚ 24.376953125 disk.nvme0n1.in‚ 11.734375% disk.nvme0n1.out‚2093.14697265625 disk.dm-0.in‚ 11.734375" disk.dm-0.out‚2093.14697265625 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 17.82421875# proc.memory.rssMB‚ 2888.83984375 cpu‚25.03283936738472 network.recv‚238124 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.87607162705773% disk./.usageGB‚1100.8068542480469$ memory_percent‚94.44842338562012 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 17.82421875 disk.drbd1777.in‚0' proc.memory.availableMB‚ 909.5703125* proc.memory.percent‚17.632079124450684 disk.rbd3.in‚0 disk.sda3.out‚ 17.82421875 disk.nvme0c0n1.in‚ 11.734375 disk.drbd1777.out‚0 network.sent‚700936 disk.sda1.in‚02Ø©KŸà‚(:” êËåÔê„ØÈ gpu.0.gpu‚69.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚46.0 gpu.0.powerWatts‚256.408' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.281600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎâÔÿ á‚(:þ êËåÔ„ý²Ë disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375 disk.rbd0.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1100.8087997436523 disk.rbd2.in‚0! disk.dm-0.out‚2192.3232421875 disk.rbd3.in‚0$ disk.nvme0n1.out‚2192.3232421875 disk.sda3.in‚0& disk.nvme0c0n1.out‚2192.3232421875 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚94.44687366485596 disk.nvme0c0n1.in‚ 11.734375 network.sent‚706423) disk./.usagePercent‚73.87620219090019 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 18.21484375 network.recv‚242136 disk.drbd1777.in‚0 disk.rbd2.out‚ 24.408203125 disk.nvme0n1.in‚ 11.734375* proc.memory.percent‚17.632079124450684 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2888.83984375 cpu‚25.033377488129126 disk.rbd1.out‚0 disk.sda.out‚ 18.21484375 disk.dm-1.out‚ 18.21484375( proc.memory.availableMB‚ 909.82421875•¡¼¬0â+ _wandbruntime‚436  _runtime‚436êð®dÆãÀ trainer/global_step‚549 % _timestamp‚1.7884379960528932e+09 _runtime‚ 436.097676552 _step‚40 ' lr-AdamW/pg1‚0.00017596793849046618 ' lr-AdamW/pg2‚0.00017596793849046618«äÊðÊäÄ trainer/global_step‚549 % _timestamp‚1.7884379960528932e+09 _runtime‚ 436.097676552 _step‚40 ' lr-AdamW/pg1‚0.00017596793849046618 ' lr-AdamW/pg2‚0.00017596793849046618(ï—¢i0å+ _wandbruntime‚436  _runtime‚436BÍeÖ€æ×O M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8784245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1006408091648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60408803629056 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1385 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60408803629056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50868348141568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 321013760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7837357 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59402421928960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59402395537408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5060185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10060763 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1009007534080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59526059948314 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60411429463040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59402421928960 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1394657548288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58521304779776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 10Ê–¿Ö] 07353445658 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8784708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59402395537408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7837351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7837351 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10060763 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60411429463040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50868376275968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10079299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1009007534080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1004755168538 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644ñûá'çÛO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8784708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59402395537408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23662166016 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1570 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7837351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7837351 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10060763 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60411429463040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50868376275968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10079299 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1009007534080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1004755168538 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3292528640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2644 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8784245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2644 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1006408091648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60408803629056 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1385 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3292528640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60408803629056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50868348141568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 321013760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1570 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7837357 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59402421928960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59402395537408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5060185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23662166016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10060763 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1009007534080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59526059948314 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60411429463040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59402421928960 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1394657548288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2644 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58521304779776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1007353445658)õàdƒ0è+ _wandbruntime‚436  _runtime‚436ÿû+Eç&éáM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60518575663104 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2639001600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59634230465254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8800480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50960764329472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 _step‚42 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52357908084736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 436.843177135 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7851593 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60521201510400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58627671382016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59510335242752 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7851593 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1385 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10097559 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17949152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3731096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58627645954048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60521201510400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 trainer/global_step‚549 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59510335242752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 % _timestamp‚1.7884379967980702e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1008240420352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7851587 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8800026 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59510361634304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1008240420352%Ñ*@ˆ%êåM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58627671382016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59510335242752 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7851593 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23662166016 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1385 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10097559 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17949152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3731096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58627645954048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2644 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60521201510400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3292528640 trainer/global_step‚549 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59510335242752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 % _timestamp‚1.7884379967980702e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1008240420352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7851587 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8800026 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59510361634304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1008240420352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60518575663104 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2639001600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59634230465254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8800480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50960764329472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1570 _step‚42 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52357908084736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2644 D>DeviceStatsMonitor.on_train_batch_end/segment.large_po‰>[Hcol.freed‚0 _runtime‚ 436.843177135 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7851593 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60521201510400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0*Zýï80ë+ _wandbruntime‚436  _runtime‚436¼TJ{PìÊ 1loss/SRqqt4/assignment_loss‚3.270596290246034 1loss/FBt4/assignment_loss‚0.43130179388182505 epoch‚0 %loss/total_loss‚81.20287566479684 % _timestamp‚1.7884379967996995e+09 1loss/SRqqt3/assignment_loss‚2.992802775823153 .loss/FRt4/detection_loss‚3.119051005352627 /loss/FRt4/assignment_loss‚6.828076080842451 1loss/SRqqt4/detection_loss‚2.2119705423404423 trainer/global_step‚549 _step‚43 0loss/FBt2/assignment_loss‚0.6202175133682127 _runtime‚ 436.844259845 /loss/FRt1/assignment_loss‚4.935419550648442 1loss/SRqqt1/assignment_loss‚2.798980104923248 1loss/SRqqt2/assignment_loss‚2.398480578511953 0loss/FBt1/assignment_loss‚0.4424605952349935 0loss/FBt3/assignment_loss‚0.3298602264399402 1loss/SRqqt1/detection_loss‚2.6566404688358305 /loss/FBt3/detection_loss‚3.8065711443915085 .loss/FBt4/detection_loss‚6.693777893270765 /loss/FRt3/assignment_loss‚6.651188203266689 0loss/SRqqt3/detection_loss‚4.582308464325392 /loss/FBt1/detection_loss‚1.7849675742584103 .loss/FBt2/detection_loss‚8.887134124835333 /loss/FRt2/detection_loss‚2.3918829153454495 /loss/FRt2/assignment_loss‚5.883583371601407 /loss/FRt1/detection_loss‚2.6954591583322594 /loss/FRt3/detection_loss‚3.0825019649096896 0loss/SRqqt2/detection_loss‚1.707643323810771òohZTíÎ 1loss/SRqqt1/assignment_loss‚2.798980104923248 1loss/SRqqt2/assignment_loss‚2.398480578511953 0loss/FBt1/assignment_loss‚0.4424605952349935 0loss/FBt3/assignment_loss‚0.3298602264399402 1loss/SRqqt1/detection_loss‚2.6566404688358305 /loss/FBt3/detection_loss‚3.8065711443915085 .loss/FBt4/detection_loss‚6.693777893270765 /loss/FRt3/assignment_loss‚6.651188203266689 0loss/SRqqt3/detection_loss‚4.582308464325392 /loss/FBt1/detection_loss‚1.7849675742584103 .loss/FBt2/detection_loss‚8.887134124835333 /loss/FRt2/detection_loss‚2.3918829153454495 /loss/FRt2/assignment_loss‚5.883583371601407 /loss/FRt1/detection_loss‚2.6954591583322594 /loss/FRt3/detection_loss‚3.0825019649096896 0loss/SRqqt2/detection_loss‚1.707643323810771 1loss/SRqqt4/assignment_loss‚3.270596290246034 1loss/FBt4/assignment_loss‚0.43130179388182505 epoch‚0 %loss/total_loss‚81.20287566479684 % _timestamp‚1.7884379967996995e+09 1loss/SRqqt3/assignment_loss‚2.992802775823153 .loss/FRt4/detection_loss‚3.119051005352627 /loss/FRt4/assignment_loss‚6.828076080842451 1loss/SRqqt4/detection_loss‚2.2119705423404423 trainer/global_step‚549 _step‚43 0loss/FBt2/assignment_loss‚0.6202175133682127 _runtime‚ 436.844259845 /loss/FRt1/assignment_loss‚4.935419550648442+ª-Y”î‚(:‰ ùËåÔÖ±™É gpu.0.gpu‚76.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.186111450195312* gpu.0.memoryAllocatedBytes‚ 24211685376 gpu.0.temp‚46.0 gpu.0.powerWatts‚240.033' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oyUîï‚(:ã ùËåÔ®ÆÌ) disk./.usagePercent‚73.87620219090019% disk./.usageGB‚1100.8087997436523& disk.nvme0c0n1.out‚2291.5185546875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.out‚19$ memory_percent‚94.44608688354492 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 11.734375 disk.dm-0.in‚ 11.734375 disk.drbd1777.out‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 909.953125 cpu‚25.034298730485457 network.recv‚246631 disk.dm-1.out‚19 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 11.734375 disk.rbd0.in‚0! disk.dm-0.out‚2291.5185546875$ disk.nvme0n1.out‚2291.5185546875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.out‚19 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2888.83984375* proc.memory.percent‚17.632079124450684 network.sent‚712733 disk.rbd1.out‚0 disk.rbd2.out‚25.63916015625uSli”ð‚(:‰ ˆÌåÔ‹ÒåÈ gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.188552856445316* gpu.0.memoryAllocatedBytes‚ 24213782528 gpu.0.temp‚46.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÜŠAñ‚(:û ˆÌåÔ‹ëÐË) disk./.usagePercent‚73.87620219090019' disk.nvme0c0n1.out‚2391.38720703125 disk.sda.out‚ 19.6015625 disk.nvme0c0n1.in‚ 11.734375 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0$ memory_percent‚94.44761276245117 disk.nvme0n1.in‚ 11.734375 disk.sda3.out‚ 19.6015625 disk.dm-1.in‚0 disk.dm-1.out‚ 19.6015625& proc.memory.availableMB‚ 909.703125 cpu‚25.034365305288688 disk.sda1.in‚0 network.sent‚762704% disk.nvme0n1.out‚2391.38720703125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375" disk.dm-0.out‚2391.38720703125 disk.rbd2.out‚26.68212890625% disk./.usageGB‚1100.8087997436523 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.632102966308594 proc.memory.rssMB‚ 2888.84375 disk.sda.in‚0 network.recv‚252596 disk.rbd2.in‚0 disk.sda2.out‚0ÎñG0ò+ _wandbruntime‚475  _runtime‚475¿ÒçÅó¿ ' lr-AdamW/pg1‚0.00019199416239670174 ' lr-AdamW/pg2‚0.00019199416239670174 trainer/global_step‚599 $ _timestamp‚1.788438035420826e+09 _runtime‚ 475.465505315 _step‚44΂aYÉôà ' lr-AdamW/pg1‚0.00019199416239670174 ' lr-AdamW/pg2‚0.00019199416239670174 trainer/global_step‚599 $ _timestamp‚1.788438035420826e+09 _runtime‚ 475.465505315 _step‚44,´ÇYX0õ+ _wandbruntime‚475  _runtime‚475löÜ'öÖO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64937223091610 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1098834170266 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1096235893146 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2645 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1521009111552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1520961948160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3294625792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9583419 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23664263168 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1571 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64799380804608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚45 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8549157 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3294625792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8549151 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8549157 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9582965 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2645 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10975815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 321013760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23664263168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2645 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64799380804608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3294625792 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1571 $ _timestamp‚1.788438035421576e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65897419745280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2645 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1098038940672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1098038940672 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64934599386522 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5519988 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10994351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23664263168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1385 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4063431 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4062980 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55487750816768 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63838363493376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10975815 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10994351 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8549157 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3294625792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8549151 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8549157 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9582965 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2645 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10975815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 321013760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23664263168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2645 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64799380804608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3294625792 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1571 $ _timestamp‚1.788438035421576e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65897419745280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2645 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1098038940672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1098038940672 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64934599386522 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5519988 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10994351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23664263168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1385 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4063431 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4062980 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55487750816768 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63838363493376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10975815 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10994351 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1571 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66007191779328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57103630732800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100663513142 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23664263168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8563393 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5529181 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2645 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64907346901504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4070041 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66009817626624 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚786 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57103706017280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 " _timestamp‚1.7884380361635e+09 _runtime‚ 476.208767148 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11012611 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9599225 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10994049 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8563393 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10994049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1098065235814 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1523491862528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8563387 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11012611 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2645 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63944730095616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes)öñå.all.peak‚ 23664263168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3294625792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2645 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4069617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55580167004672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2645 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1099871269376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1099871269376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65045393608758 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3294625792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64907346901504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8563387 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9598798 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23664263168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1102470725120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64907320509952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64907320509952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5529184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55580138870272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8563393 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10994049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1098065235814 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1523491862528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8563387 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11012611 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2645 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63944730095616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23664263168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3294625792 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2645 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4069617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55580167004672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2645 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1099871269376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1099871269376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65045393608758 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3294625792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64907346901504 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8563387 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9598798 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23664263168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1102470725120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64907320509952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64907320509952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5529184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55580138870272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1571 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66007191779328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57103630732800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 321013760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100663513142 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23664263168 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8563393 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5529181 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2645 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64907346901504 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4070041 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66009817626624 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚786 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57103706017280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 " _timestamp‚1.7884380361635e+09 _runtime‚ 476.208767148 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11012611 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9599225 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10994049 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23664263168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3294625792 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1571 trainer/global_step‚649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1192269232128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5979788 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9260957 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1647024766976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69155447634944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2647390208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚71388661695488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2645 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70196392463360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚788 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71386035861504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 326393344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23664263168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚ 514.732564894 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9260951 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70343138824730 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11909403 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60107181626368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9260957 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4400101 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70345762529818 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _step‚49 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71386035861504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70196366071808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23664263168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3294625792 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1571 trainer/global_step‚649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1192269232128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5979788 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9260957 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1647024766976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69155447634944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2647390208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚71388661695488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2645 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70196392463360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚788 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71386035861504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 326393344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23664263168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚ 514.732564894 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9260951 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70343138824730 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11909403 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60107181626368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9260957 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4400101 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70345762529818 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _step‚49 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71386035861504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70196366071808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3294625792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884380754286814e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23664263168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1191502118400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4406780 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23664263168 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚71498433742848 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71495807895552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1387 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1192144237750 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4407232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70304332168704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2645 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9275187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1649590557696 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 _step‚50 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3294625792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2645 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10395761 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60199541545472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1571 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11927663 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11909101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61849082855936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1191502118400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9275187 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70453933046966 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2647390208 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1189545960422 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2645 _runtime‚ 515.473950369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9275193 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21202856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚649 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9275193 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10396216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0µN€í&ŒçM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11927663 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11909101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61849082855936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1191502118400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9275187 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70453933046966 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2647390208 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1189545960422 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2645 _runtime‚ 515.473950369 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9275193 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21202856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚649 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9275193 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10396216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1571 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1571 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1194101574144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69261788809216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3294625792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2645 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3294625792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884380754286814e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23664263168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1191502118400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4406780 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23664263168 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚71498433742848 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71495807895552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1387 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1192144237750 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4407232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70304332168704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2645 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9275187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1649590557696 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 _step‚50 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3294625792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2645 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10395761 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60199541545472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1571 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚60927Øçª0+ _wandbruntime‚515  _runtime‚515ÿ¸nÔTŽÎ /loss/FBt2/detection_loss‚7.0179743528366085 _step‚51 2loss/SRqqt3/assignment_loss‚1.6367001712503957 0loss/FBt4/assignment_loss‚0.6415013409714447 .loss/FRt2/detection_loss‚2.376639852909879 _runtime‚ 515.475506582 /loss/FRt3/assignment_loss‚6.698689404655905 /loss/FRt1/assignment_loss‚4.488936707377434 /loss/FRt4/assignment_loss‚5.653881971226182 1loss/FBt1/assignment_loss‚0.32753252358071205 0loss/SRqqt3/detection_loss‚2.650858355004613 % _timestamp‚1.7884380754309216e+09 %loss/total_loss‚74.55866379978595 /loss/FRt2/assignment_loss‚5.343896536393599 1loss/FBt3/assignment_loss‚0.27192968446070864 .loss/FRt1/detection_loss‚2.126285716192797 1loss/SRqqt1/detection_loss‚1.7270293828489176 1loss/SRqqt4/detection_loss‚3.8771154512961705 .loss/FBt3/detection_loss‚5.294575396180153 epoch‚0 1loss/SRqqt2/detection_loss‚1.9152722722717694 .loss/FBt4/detection_loss‚5.992107053575818 1loss/SRqqt4/assignment_loss‚2.860822852452596 /loss/FRt3/detection_loss‚3.0021970412310433 .loss/FRt4/detection_loss‚3.004821096741876 trainer/global_step‚649 2loss/SRqqt1/assignment_loss‚2.7615939022889777 2loss/SRqqt2/assignment_loss‚2.6070058490548815 1loss/FBt2/assignment_loss‚0.39691333994269373 /loss/FBt1/detection_loss‚1.8843835450407715ÞXÒ /loss/FRt4/assignment_loss‚5.653881971226182 1loss/FBt1/assignment_loss‚0.32753252358071205 0loss/SRqqt3/detection_loss‚2.650858355004613 % _timestamp‚1.7884380754309216e+09 %loss/total_loss‚74.55866379978595 /loss/FRt2/assignment_loss‚5.343896536393599 1loss/FBt3/assignment_loss‚0.27192968446070864 .loss/FRt1/detection_loss‚2.126285716192797 1loss/SRqqt1/detection_loss‚1.7270293828489176 1loss/SRqqt4/detection_loss‚3.8771154512961705 .loss/FBt3/detection_loss‚5.294575396180153 epoch‚0 1loss/SRqqt2/detection_loss‚1.9152722722717694 .loss/FBt4/detection_loss‚5.992107053575818 1loss/SRqqt4/assignment_loss‚2.860822852452596 /loss/FRt3/detection_loss‚3.0021970412310433 .loss/FRt4/detection_loss‚3.004821096741876 trainer/global_step‚649 2loss/SRqqt1/assignment_loss‚2.7615939022889777 2loss/SRqqt2/assignment_loss‚2.6070058490548815 1loss/FBt2/assignment_loss‚0.39691333994269373 /loss/FBt1/detection_loss‚1.8843835450407715 /loss/FBt2/detection_loss‚7.0179743528366085 _step‚51 2loss/SRqqt3/assignment_loss‚1.6367001712503957 0loss/FBt4/assignment_loss‚0.6415013409714447 .loss/FRt2/detection_loss‚2.376639852909879 _runtime‚ 515.475506582 /loss/FRt3/assignment_loss‚6.698689404655905 /loss/FRt1/assignment_loss‚4.4889367073774343´@RÚ’‚(:‡ ÄÌåÔ‘í˜É gpu.0.gpu‚33.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.188552856445316* gpu.0.memoryAllocatedBytes‚ 24213782528 gpu.0.temp‚42.0 gpu.0.powerWatts‚92.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l!šf ‘‚(:!ª§þ ÄÌåÔ®½ÓË disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 11.734375$ disk.nvme0c0n1.out‚ 2690.01953125* proc.memory.percent‚17.632150650024414 disk.rbd1.out‚ 0.1484375 cpu‚25.035609685514085 disk.sda3.out‚ 21.9921875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 11.734375 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 11.734375 disk.rbd2.in‚0) disk./.usagePercent‚73.87621857538238 disk.rbd3.out‚0 disk.sda3.in‚0 network.sent‚830858% disk./.usageGB‚1100.8090438842773" disk.nvme0n1.out‚ 2690.01953125 disk.dm-0.out‚ 2690.01953125 disk.rbd1.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2888.8515625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚ 21.9921875$ memory_percent‚94.44668292999268( proc.memory.availableMB‚ 909.85546875 disk.rbd0.out‚0 disk.dm-1.out‚ 21.9921875 disk.drbd1777.out‚0 disk.rbd2.out‚30.13134765625 network.recv‚272249 disk.sda1.in‚0Ówÿ\”’‚(:‰ ÓÌåÔº™ŒÉ gpu.0.gpu‚80.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚28.188552856445316* gpu.0.memoryAllocatedBytes‚ 24213782528 gpu.0.temp‚46.0 gpu.0.powerWatts‚218.151' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.6302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ý“‚(:ò ÓÌåÔÇíÁË network.recv‚278522 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.out‚2696.625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.1806640625* proc.memory.percent‚17.632150650024414 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 11.734375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.out‚2696.625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 30.2919921875 disk.sda.out‚ 22.88671875) disk./.usagePercent‚73.87661385101525% disk./.usageGB‚1100.8149337768555 disk.nvme0c0n1.in‚ 11.734375 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚94.44687366485596 network.sent‚881290 disk.drbd1777.out‚0 disk.nvme0n1.out‚2696.625( proc.memory.availableMB‚ 909.82421875" proc.memory.rssMB‚ 2888.8515625 cpu‚25.0360028837556 disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375 disk.dm-1.in‚0 disk.dm-1.out‚ 22.88671875 disk.sda3.out‚ 22.88671875 disk.sda1.in‚0 disk.sda1.out‚0~³õ•0”+ _wandbruntime‚554  _runtime‚554¹ÕšIÕ½ & lr-AdamW/pg1‚0.0002240466102091728 & lr-AdamW/pg2‚0.0002240466102091728 trainer/global_step‚699 % _timestamp‚1.7884381141526666e+09 _runtime‚ 554.19730704 _step‚52ÆG ”Ç–Á & lr-AdamW/pg1‚0.0002240466102091728 & lr-AdamW/pg2‚0.0002240466102091728 trainer/global_step‚699 % _timestamp‚1.7884381141526666e+09 _runtime‚ 554.19730704 _step‚524Ñ®~.0—+ _wandbruntime‚554  _runtime‚5548–Kÿß'˜ÙO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 333804544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23664263168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75593351339008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1773015697920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2645 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1772966437376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2645 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1279197342362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11177640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6439585 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23664263168 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76874651977728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4737576 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1571 trainer/global_step‚699 _runtime‚ 554.198227866 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74472480920576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9972757 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64726584301568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3294625792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22797212 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11177161 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12824455 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76877277811712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚53 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1283900081152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75751678262938 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74472506348544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75593351339008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3294625792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3294625792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12805919 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1389 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1571 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12824455 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75593377730560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1281300638720 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9972751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4738052 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1281795619482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76874651977728 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76877277811712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75754301968026 % _timestamp‚1.7884381141533902e+09ï:f`ã'™ÝO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75593351339008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3294625792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3294625792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12805919 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1389 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1571 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12824455 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75593377730560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1281300638720 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9972751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4738052 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1281795619482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76874651977728 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76877277811712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75754301968026 % _timestamp‚1.7884381141533902e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 333804544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23664263168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75593351339008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1773015697920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2645 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1772966437376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2645 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1279197342362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11177640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6439585 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23664263168 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76874651977728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4737576 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1571 trainer/global_step‚699 _runtime‚ 554.198227866 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74472480920576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9972757 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64726584301568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3294625792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22797212 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11177161 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12824455 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76877277811712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚53 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1283900081152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75751678262938 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74472506348544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 25982771205ËÙ¸Ä0š+ _wandbruntime‚554  _runtime‚554À_,ýF$›ãM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12842715 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66594427488768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 trainer/global_step‚699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11193533 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74578822094848 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76984424011776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9986993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884381148927627e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9986987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6448784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64818944220672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2657875968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75859848779878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51344384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11193056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1285732423168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚ 554.937931071 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1389 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4744275 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9986987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12824153 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76987049859072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1283132967424 ]JDeviceStatsMonitor.on_train_batch_end/requí‘ýd£ested_bytes.large_pool.allocated‚74578847522816 _step‚54 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75701317435904 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1283624962358 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76984424011776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9986993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884381148927627e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9986987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6448784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64818944220672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2657875968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75859848779878 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚51344384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11193056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1285732423168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚ 554.937931071 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1389 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4744275 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9986987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12824153 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76987049859072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1283132967424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74578847522816 _step‚54 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75701317435904 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1283624962358 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O¬”¤‚(:‰ €ÍåÔØØêÈ gpu.0.gpu‚70.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚246.201' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤gø¥‚(:û €ÍåÔ¼´†Ë$ disk.nvme0n1.out‚2994.6806640625 disk.rbd2.out‚ 33.61328125 disk.sda2.in‚0) disk./.usagePercent‚73.87664815602484& disk.nvme0c0n1.out‚2994.6806640625 disk.sda3.out‚24.6875 disk.drbd1777.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚2994.6806640625 disk.sda.out‚24.6875 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚294209 disk.rbd2.in‚0( proc.memory.availableMB‚ 910.24609375 disk.rbd3.in‚0 disk.dm-1.out‚24.6875 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 11.734375 disk.dm-0.in‚ 11.734375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.24365234375* proc.memory.percent‚17.632150650024414 disk.sda1.in‚0$ memory_percent‚94.44429874420166" proc.memory.rssMB‚ 2888.8515625 cpu‚25.036663325745717 proc.cpu.threads‚17$ disk./.usageGB‚1100.815444946289 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 11.734375 network.sent‚943810 disk.dm-1.in‚0 disk.sda.in‚0„2°0¦+ _wandbruntime‚593  _runtime‚593.ħ¾ % _timestamp‚1.7884381531685474e+09 _runtime‚ 593.213211199 _step‚56 & lr-AdamW/pg1‚0.0002400728341154083 & lr-AdamW/pg2‚0.0002400728341154083 trainer/global_step‚749Ñ®QWȨ & lr-AdamW/pg1‚0.0002400728341154083 & lr-AdamW/pg2‚0.0002400728341154083 trainer/global_step‚749 % _timestamp‚1.7884381531685474e+09 _runtime‚ 593.213211199 _step‚568TS„·0©+ _wandbruntime‚593  _runtime‚593ë!†ã'ªÝO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370678066970 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚57 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5076930 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1898602767360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1372931487744 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5076464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80990362997760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82363268093952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _runtime‚ 593.214222184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1898553506816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82365893927936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71244512349184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82363268093952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6899385 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10684557 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13739507 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82365893927936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1898553506816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82365893927936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71244512349184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82363268093952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6899385 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10684557 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13739507 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82365893927936 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10698787 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13739205 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1377363272192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 % _timestamp‚1.7884381539139812e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1375105686966 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 _runtime‚ 593.959084534 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13739205 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11991749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6908584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10698793 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11992202 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71339493828096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10698793 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5083168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5083618 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372507409638 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901118797824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6908581 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81098302703104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81098276311552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1374763816448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81098302703104ÿßBLñ&®ëM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372507409638 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901118797824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6908581 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81098302703104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24437992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81098276311552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1374763816448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81098302703104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1377363272192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10698787 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 trainer/global_step‚749 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82475665975296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81098276311552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79895880808448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82475665975296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚58 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13757767 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10698787 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13739205 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1377363272192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 % _timestamp‚1.7884381539139812e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1375105686966 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 _runtime‚ 593.959084534 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13739205 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11991749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6908584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10698793 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11992202 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71339493828096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10698793 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5083168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5083618 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450:Knã0¯+ _wandbruntime‚593  _runtime‚5937_’pV°Ð trainer/global_step‚749 _step‚59 0loss/FBt2/assignment_loss‚0.9225909795894984 _runtime‚ 593.960254429 /loss/FRt2/detection_loss‚2.3681192544513734 0loss/FRt1/assignment_loss‚3.6275505547253593 /loss/FRt3/assignment_loss‚6.424427164925469 0loss/FBt4/assignment_loss‚0.5426512377982059 1loss/SRqqt2/detection_loss‚1.6272378369599958 /loss/FBt2/detection_loss‚3.9501969539202175 % _timestamp‚1.7884381539156673e+09 1loss/FBt3/assignment_loss‚0.35800621993839743 0loss/SRqqt1/detection_loss‚1.334106196080231 /loss/FBt4/detection_loss‚2.2866700731187675 2loss/SRqqt2/assignment_loss‚2.4867885459827472 %loss/total_loss‚64.52495914618228 0loss/FRt2/assignment_loss‚4.4678065612398346 1loss/SRqqt1/assignment_loss‚2.616127662360668 /loss/FRt3/detection_loss‚2.9389034112294516 1loss/SRqqt3/detection_loss‚2.5288630316131995 1loss/SRqqt4/detection_loss‚3.2837092358133066 1loss/SRqqt3/assignment_loss‚2.407035396287316 2loss/SRqqt4/assignment_loss‚3.3483395006345664 /loss/FRt4/detection_loss‚3.2907702115274247 /loss/FBt1/detection_loss‚0.8015212466493835 .loss/FBt3/detection_loss‚3.874657228589058 epoch‚0 .loss/FRt1/detection_loss‚2.421465624053523 /loss/FRt4/assignment_loss‚6.326660540796095 0loss/FBt1/assignment_loss‚0.2907544778981934å*¥OZ±Ô trainer/global_step‚749 _step‚59 0loss/FBt2/assignment_loss‚0.9225909795894984 _runtime‚ 593.960254429 /loss/FRt2/detection_loss‚2.3681192544513734 0loss/FRt1/assignment_loss‚3.6275505547253593 /loss/FRt3/assignment_loss‚6.424427164925469 0loss/FBt4/assignment_loss‚0.5426512377982059 1loss/SRqqt2/detection_loss‚1.6272378369599958 /loss/FBt2/detection_loss‚3.9501969539202175 % _timestamp‚1.7884381539156673e+09 1loss/FBt3/assignment_loss‚0.35800621993839743 0loss/SRqqt1/detection_loss‚1.334106196080231 /loss/FBt4/detection_loss‚2.2866700731187675 2loss/SRqqt2/assignment_loss‚2.4867885459827472 %loss/total_loss‚64.52495914618228 0loss/FRt2/assignment_loss‚4.4678065612398346 1loss/SRqqt1/assignment_loss‚2.616127662360668 /loss/FRt3/detection_loss‚2.9389034112294516 1loss/SRqqt3/detection_loss‚2.5288630316131995 1loss/SRqqt4/detection_loss‚3.2837092358133066 1loss/SRqqt3/assignment_loss‚2.407035396287316 2loss/SRqqt4/assignment_loss‚3.3483395006345664 /loss/FRt4/detection_loss‚3.2907702115274247 /loss/FBt1/detection_loss‚0.8015212466493835 .loss/FBt3/detection_loss‚3.874657228589058 epoch‚0 .loss/FRt1/detection_loss‚2.421465624053523 /loss/FRt4/assignment_loss‚6.326660540796095 0loss/FBt1/assignment_loss‚0.2907544778981934;Š“ñvŸ²‚(:” ÍåÔ°ŽŽÉ gpu.0.gpu‚64.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚253.755' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.751000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êvºt³‚(:ý ÍåÔÈý¹Ë disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚2995.0478515625 cpu‚25.03694910578074 network.sent‚950210 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.dm-0.in‚ 11.734375 disk.dm-1.out‚ 25.3515625# memory_percent‚94.4476842880249" proc.memory.rssMB‚ 2888.8515625 network.recv‚298766 disk.sda2.in‚0 disk.nvme0n1.in‚ 11.734375 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 11.734375( proc.memory.availableMB‚ 909.69140625 disk.drbd1777.in‚0 disk.rbd2.out‚ 33.69140625 disk.sda.out‚ 25.3515625 disk.sda1.out‚0$ disk.nvme0n1.out‚2995.0478515625" disk./.usageGB‚1100.8154296875 disk.sda.in‚0* proc.memory.percent‚17.632150650024414 disk.sda1.in‚0 disk.sda3.out‚ 25.3515625 disk.rbd2.in‚0& disk.nvme0c0n1.out‚2995.0478515625 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.dm-1.in‚0( disk./.usagePercent‚73.8766471319947䜒©”´‚(:‰ žÍåÔïΟÉ gpu.0.gpu‚93.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚213.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.7524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YU¬ûµ‚(:ð žÍåÔ çÁË network.sent‚999930( disk./.usagePercent‚73.8766471319947 disk.dm-0.out‚3094.228515625 disk.nvme0c0n1.in‚ 11.734375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 cpu‚25.037248838522338 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 909.69140625" proc.memory.rssMB‚ 2888.8515625 disk.rbd2.out‚ 34.796875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.632150650024414 proc.cpu.threads‚17 disk.sda.out‚25.9375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 11.734375" disk./.usageGB‚1100.8154296875 disk.dm-0.in‚ 11.734375 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚3094.228515625 network.recv‚304472 disk.sda1.out‚0# memory_percent‚94.4476842880249 disk.dm-1.out‚25.9375# disk.nvme0n1.out‚3094.228515625 disk.sda3.out‚25.9375ô.=ƒ”¶‚(:‰ ­ÍåÔÛ”É gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚258.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.7324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}‹¬Dc·‚(:‹ ­ÍåÔ³‡­Ë proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625! disk.dm-0.out‚3193.3701171875 network.sent‚ 1.005884e+06 network.recv‚308720 disk.sda3.out‚ 26.68359375 disk.rbd3.out‚0" proc.memory.rssMB‚ 2888.8515625 disk.nvme0n1.in‚ 11.734375$ memory_percent‚94.44549083709717" disk./.usageGB‚1100.8154296875 disk.rbd0.in‚0 disk.dm-1.out‚ 26.68359375 disk.sda.out‚ 26.68359375 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚3193.3701171875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.‚$‹™³in‚0 disk.sda.in‚0* proc.memory.percent‚17.632150650024414( disk./.usagePercent‚73.8766471319947$ disk.nvme0n1.out‚3193.3701171875 disk.nvme0c0n1.in‚ 11.734375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 910.05078125 disk.rbd3.in‚0 cpu‚25.037172753295625 disk.sda2.in‚0 disk.dm-0.in‚ 11.734375 disk.rbd2.out‚36.02392578125 disk.sda1.in‚0ä5Ÿ‹0¸+ _wandbruntime‚632  _runtime‚632®¿KĹ¾ trainer/global_step‚799 % _timestamp‚1.7884381926707635e+09 _runtime‚ 632.715421263 _step‚60 & lr-AdamW/pg1‚0.0002560990580216439 & lr-AdamW/pg2‚0.0002560990580216439Ô¼r³ÈºÂ _step‚60 & lr-AdamW/pg1‚0.0002560990580216439 & lr-AdamW/pg2‚0.0002560990580216439 trainer/global_step‚799 % _timestamp‚1.7884381926707635e+09 _runtime‚ 632.715421263DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75989643228672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79492096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11396351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2024333068800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚87854510044160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86387348264960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1464562336768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5413936 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87851884210176 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86387321873408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73965361517568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚87851884210176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14636023 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75989722720768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2666264576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86387321873408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12773121 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14636023 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7359188 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73965389651968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86571380844442 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 % _timestamp‚1.7884381926715176e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86387348264960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51357696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14654559 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚87854510044160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86387348264960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1464562336768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5413936 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87851884210176 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86387321873408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73965361517568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚87851884210176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14636023 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75989722720768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2666264576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86387321873408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12773121 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14636023 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7359188 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73965389651968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86571380844442 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 % _timestamp‚1.7884381926715176e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86387348264960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚51357696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14654559 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75989643228672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚79492096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11396351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2024333068800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074=5¸‡0¾+ _wandbruntime‚633  _runtime‚633q6ží&¿çM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 % _timestamp‚1.7884381934135652e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7368381 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 _step‚62 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87964282091520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11410587 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12789398 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5421014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 _runtime‚ 633.458931641 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49247232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2026783784448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2026833031680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86495287970304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚77381632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74057749571072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14654257 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1468994121216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14654257 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1468994121216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14672819 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11410587 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85212964950016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86495261578752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1466394665472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74057777705472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12788943 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11410593 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11410593 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86495287970304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14672819 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 trainer/global_step‚799 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86676927656294GÕpŒ4ÀëM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26064844 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14654257 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1468994121216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14654257 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1468994121216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14672819 =7DeviceStatsMonitor.on_trþ´î½!ain_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11410587 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85212964950016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86495261578752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1466394665472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74057777705472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12788943 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11410593 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11410593 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86495287970304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14672819 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 trainer/global_step‚799 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86676927656294 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 % _timestamp‚1.7884381934135652e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7368381 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 _step‚62 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87964282091520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11410587 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12789398 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5421014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 _runtime‚ 633.458931641 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49247232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2026783784448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2026833031680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86495287970304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚77381632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74057749571072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FŽ†ÑX0Á+ _wandbruntime‚633  _runtime‚633¢{ñUÂÏ 2loss/SRqqt3/assignment_loss‚2.9791783012192825 /loss/FBt1/detection_loss‚1.2839161006025508 .loss/FBt3/detection_loss‚2.843155139684677 epoch‚0 trainer/global_step‚799 _runtime‚ 633.460038797 /loss/FRt1/detection_loss‚2.1173043976537884 1loss/FBt3/assignment_loss‚0.36833252378112774 0loss/FBt4/assignment_loss‚0.3992231455654007 1loss/SRqqt2/detection_loss‚1.8823286425608856 /loss/FRt2/assignment_loss‚4.741011750941374 /loss/FRt2/detection_loss‚2.4664047883481395 .loss/FRt3/detection_loss‚2.746487048956064 1loss/SRqqt3/detection_loss‚2.9376165887405135 0loss/SRqqt4/detection_loss‚2.386907847304093 /loss/FBt4/detection_loss‚2.6843008124915286 0loss/FRt1/assignment_loss‚3.5969384908676147 /loss/FRt4/assignment_loss‚5.024137826192947 2loss/SRqqt2/assignment_loss‚2.5950674084516674 1loss/SRqqt4/assignment_loss‚2.395069724635074 .loss/FRt4/detection_loss‚2.483229492391859 1loss/SRqqt1/assignment_loss‚2.334198568318341 /loss/FRt3/assignment_loss‚6.890796514657827 .loss/FBt2/detection_loss‚4.371271632611752 % _timestamp‚1.7884381934154532e+09 1loss/FBt1/assignment_loss‚0.21310738390944686 0loss/FBt2/assignment_loss‚0.4216765118762851 1loss/SRqqt1/detection_loss‚1.4372677604693014 &loss/total_loss‚61.598928402231536 _step‚63UvYÃÓ 1loss/SRqqt1/assignment_loss‚2.334198568318341 /loss/FRt3/assignment_loss‚6.890796514657827 .loss/FBt2/detection_loss‚4.371271632611752 % _timestamp‚1.7884381934154532e+09 1loss/FBt1/assignment_loss‚0.21310738390944686 0loss/FBt2/assignment_loss‚0.4216765118762851 1loss/SRqqt1/detection_loss‚1.4372677604693014 &loss/total_loss‚61.598928402231536 _step‚63 2loss/SRqqt3/assignment_loss‚2.9791783012192825 /loss/FBt1/detection_loss‚1.2839161006025508 .loss/FBt3/detection_loss‚2.843155139684677 epoch‚0 trainer/global_step‚799 _runtime‚ 633.460038797 /loss/FRt1/detection_loss‚2.1173043976537884 1loss/FBt3/assignment_loss‚0.36833252378112774 0loss/FBt4/assignment_loss‚0.3992231455654007 1loss/SRqqt2/detection_loss‚1.8823286425608856 /loss/FRt2/assignment_loss‚4.741011750941374 /loss/FRt2/detection_loss‚2.4664047883481395 .loss/FRt3/detection_loss‚2.746487048956064 1loss/SRqqt3/detection_loss‚2.9376165887405135 0loss/SRqqt4/detection_loss‚2.386907847304093 /loss/FBt4/detection_loss‚2.6843008124915286 0loss/FRt1/assignment_loss‚3.5969384908676147 /loss/FRt4/assignment_loss‚5.024137826192947 2loss/SRqqt2/assignment_loss‚2.5950674084516674 1loss/SRqqt4/assignment_loss‚2.395069724635074 .loss/FRt4/detection_loss‚2.483229492391859?qæ+ŸÄ‚(:” ¼ÍåÔ Î™É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.742' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.548399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒâê6Å‚(:Ž ¼ÍåÔÛ¨ÒË disk.rbd3.in‚0' disk.nvme0c0n1.out‚3292.80517578125 network.sent‚ 1.011706e+06 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 909.9921875* proc.memory.percent‚17.632150650024414 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚94.44584846496582 cpu‚25.037436150673756) disk./.usagePercent‚73.87664738800224% disk./.usageGB‚1100.8154335021973 disk.dm-1.in‚0 disk.dm-1.out‚ 27.0546875 disk.sda.in‚0 disk.nvme0n1.in‚ 11.734375 disk.sda1.in‚0% disk.nvme0n1.out‚3292.80517578125 disk.sda3.out‚ 27.0546875 disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375" disk.dm-0.out‚3292.80517578125 network.recv‚313193 disk.rbd2.out‚37.12939453125 disk.sda.out‚ 27.0546875 disk.nvme0c0n1.in‚ 11.734375" proc.memory.rssMB‚ 2888.8515625 proc.cpu.threads‚17 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0F":¬“Æ‚(:ˆ ËÍåÔª´¢É gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚237.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.415 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q™3Ç‚(:‘ ËÍåÔ—ؼË network.sent‚ 1.062141e+06 disk.rbd3.in‚0' proc.memory.availableMB‚ 909.7578125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd2.out‚37.39501953125 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚3293.16162109375 disk.nvme0c0n1.in‚ 11.734375 disk.drbd1777.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2888.8515625 cpu‚25.038087884843183% disk./.usageGB‚1100.8151893615723 disk.rbd3.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚3293.16162109375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 11.734375 disk.drbd1777.out‚0* proc.memory.percent‚17.632150650024414 network.recv‚319404) disk./.usagePercent‚73.87663100352005 disk.nvme0n1.in‚ 11.734375' disk.nvme0c0n1.out‚3293.16162109375 disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 27.77734375 disk.sda1.out‚0 disk.sda3.out‚ 27.77734375 disk.dm-1.out‚ 27.77734375$ memory_percent‚94.44727897644043 disk.rbd0.out‚0ÒDæF0È+ _wandbruntime‚672  _runtime‚672™eNÄɾ & lr-AdamW/pg1‚0.0002721252819278794 & lr-AdamW/pg2‚0.0002721252819278794 trainer/global_step‚849 % _timestamp‚1.7884382320356238e+09 _runtime‚ 672.080430746 _step‚64ÆnnÈÊ & lr-AdamW/pg1‚0.0002721252819278794 & lr-AdamW/pg2‚0.0002721252819278794 trainer/global_step‚849 % _timestamp‚1.7884382320356238e+09 _runtime‚ 672.080430746 _step‚64@}Ymý0Ë+ _wandbruntime‚672  _runtime‚672‡÷½ã'ÌÝO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1556193185792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78584792327168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _runtime‚ 672.081912679 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5752085 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90423657061376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15551075 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13571527 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93340500326400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15569611 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90423682489344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12108151 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5752539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2150536799232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 trainer/global_step‚849 % _timestamp‚1.7884382320365934e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27677768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7818985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91784307140608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93340500326400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12108157 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13571070 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93343126160384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12108157 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _runtime‚ 672.081912679 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5752085 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90423657061376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15551075 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13571527 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93340500326400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15569611 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90423682489344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12108151 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5752539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2150536799232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 trainer/global_step‚849 % _timestamp‚1.7884382320365934e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27677768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7818985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91784307140608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93340500326400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12108157 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13571070 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93343126160384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12108157 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1556193185792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78584792327168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 _runtime‚ 672.831175544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2153015066624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 # _timestamp‚1.78843823278603e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13587398 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91892273237504 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1558067136182 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80830240500224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12122387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1560624970240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80830167312896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2153060119552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12122393 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91892246845952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1560624970240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5758789 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13586970 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91892246845952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12122387 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78677180380672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1558067136182 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80830240500224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12122387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1560624970240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80830167312896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2153060119552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12122393 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91892246845952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1560624970240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5758789 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13586970 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91892246845952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12122387 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78677180380672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 _runtime‚ 672.831175544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2153015066624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 # _timestamp‚1.78843823278603e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13587398 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91892273237504B±ó0Ñ+ _wandbruntime‚672  _runtime‚672ÞVöIOÒÉ $loss/total_loss‚61.9522797493359 epoch‚0 -loss/FRt4/assignment_loss‚5.0226452035724 2loss/SRqqt3/assignment_loss‚2.6080104327201843 1loss/SRqqt1/assignment_loss‚2.562901780245796 2loss/SRqqt2/assignment_loss‚2.1714294620098604 2loss/SRqqt4/assignment_loss‚2.1727054182327157 .loss/FBt1/detection_loss‚1.360871445542956 .loss/FBt3/detection_loss‚4.603991703553633 _runtime‚ 672.832670862 _step‚67 1loss/FBt1/assignment_loss‚0.31834875818723657 1loss/SRqqt2/detection_loss‚1.7454835597065188 /loss/FRt1/detection_loss‚2.1076716828872177 % _timestamp‚1.7884382327880397e+09 .loss/FRt3/detection_loss‚2.612609096935817 1loss/SRqqt1/detection_loss‚1.6768435274088194 trainer/global_step‚849 0loss/FRt1/assignment_loss‚2.9795381286564995 /loss/FRt2/assignment_loss‚4.098259563348731 .loss/FRt3/assignment_loss‚6.34599460874285 0loss/FBt2/assignment_loss‚0.5560777969658375 0loss/FBt4/assignment_loss‚0.5759311308824893 .loss/FBt2/detection_loss‚4.717334474836077 /loss/FBt4/detection_loss‚2.9462930553633235 1loss/FBt3/assignment_loss‚0.38497143983835774 0loss/SRqqt3/detection_loss‚3.144916839599609 /loss/FRt2/detection_loss‚2.4971553628541985 .loss/FRt4/detection_loss‚2.489973911136951 0loss/SRqqt4/detection_loss‚2.252321366107825šq¤ÿSÓÍ /loss/FRt2/detection_loss‚2.4971553628541985 .loss/FRt4/detection_loss‚2.489973911136951 0loss/SRqqt4/detection_loss‚2.252321366107825 $loss/total_loss‚61.9522797493359 epoch‚0 -loss/FRt4/assignment_loss‚5.0226452035724 2loss/SRqqt3/assignment_loss‚2.6080104327201843 1loss/SRqqt1/assignment_loss‚2.562901780245796 2loss/SRqqt2/assignment_loss‚2.1714294620098604 2loss/SRqqt4/assignment_loss‚2.1727054182327157 .loss/FBt1/detection_loss‚1.360871445542956 .loss/FBt3/detection_loss‚4.603991703553633 _runtime‚ 672.832670862 _step‚67 1loss/FBt1/assignment_loss‚0.31834875818723657 1loss/SRqqt2/detection_loss‚1.7454835597065188 /loss/FRt1/detection_loss‚2.1076716828872177 % _timestamp‚1.7884382327880397e+09 .loss/FRt3/detection_loss‚2.612609096935817 1loss/SRqqt1/detection_loss‚1.6768435274088194 trainer/global_step‚849 0loss/FRt1/assignment_loss‚2.9795381286564995 /loss/FRt2/assignment_loss‚4.098259563348731 .loss/FRt3/assignment_loss‚6.34599460874285 0loss/FBt2/assignment_loss‚0.5560777969658375 0loss/FBt4/assignment_loss‚0.5759311308824893 .loss/FBt2/detection_loss‚4.717334474836077 /loss/FBt4/detection_loss‚2.9462930553633235 1loss/FBt3/assignment_loss‚0.38497143983835774 0loss/SRqqt3/detection_loss‚3.144916839599609C’mt”Ô‚(:‰ ÚÍåÔؽëÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚247.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b'¿'Õ‚(: ÚÍåÔšºÚË disk.dm-1.in‚0" proc.memory.rssMB‚ 2888.8515625 cpu‚25.038325861865204 network.sent‚ 1.067325e+06) disk./.usagePercent‚73.87664021979128% disk./.usageGB‚1100.8153266906738 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚3392.5400390625 disk.sda.out‚ 28.25390625 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 11.734375 disk.drbd1777.out‚0* proc.memory.percent‚17.632150650024414 proc.cpu.threads‚17 disk.rbd2.out‚38.50048828125$ disk.nvme0n1.out‚3392.5400390625 disk.sda3.out‚ 28.25390625 disk.drbd1777.in‚0 network.recv‚323208 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 28.25390625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 11.734375" memory_percent‚94.449782371521 disk.rbd3.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3392.5400390625 disk.dm-0.in‚ 11.734375( proc.memory.availableMB‚ 909.34765625 disk.rbd1.out‚ 0.306640625ˆW•°”Ö‚(:‰ éÍåÔ°ÎßÈ gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À¾oÏׂ(:‹ éÍåÔæ–ÙË disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 28.9765625 disk.sda3.out‚ 28.9765625 disk.dm-0.in‚ 11.734375 cpu‚25.03787322775611 proc.cpu.threads‚17) disk./.usagePercent‚73.87661743512072 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚3491.94580078125 disk.sda3.in‚0 disk.dm-1.out‚ 28.9765625 network.sent‚ 1.116996e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚3491.94580078125* proc.memory.percent‚17.632150650024414% disk./.usageGB‚1100.8149871826172 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚39.65283203125 disk.sda.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚3491.94580078125 disk.dm-1.in‚0% proc.memory.availableMB‚ 909.03125 network.recv‚328766 disk.nvme0n1.in‚ 11.734375" proc.memory.rssMB‚ 2888.8515625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 11.734375$ memory_percent‚94.45171356201172 disk.drbd1777.in‚0ÃßÇ|žØ‚(:“ øÍåÔû¢†É gpu.0.gpu‚68.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚259.417' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.88339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯×a¹Ù‚(:‹ øÍåÔ–ð…Ë$ memory_percent‚94.44732666015625" proc.memory.availableMB‚909.75* proc.memory.percent‚17.632198333740234 disk.sda2.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚333160 disk.sda1.out‚0! proc.memory.rssMB‚ 2888.859375 cpu‚25.038167066016605 network.sent‚ 1.122883e+06) disk./.usagePercent‚73.87661743512072 disk.sda.out‚ 29.44921875% disk./.usageGB‚1100.8149871826172 disk.sda3.out‚ 29.44921875' disk.nvme0c0n1.out‚3591.56787109375" disk.dm-0.out‚3591.56787109375 disk.dm-1.in‚0 disk.dm-1.out‚ 29.44921875 disk.rbd1.out‚ 0.306640625 disk.nvme0n1.in‚ 11.734375 disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375 disk.sda.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚3591.56787109375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚39.68408203125 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 11.734375 disk.drbd1777.out‚0äÿÃ0Ú+ _wandbruntime‚711  _runtime‚711IG†ÄÛ¾ _runtime‚ 711.239023318 _step‚68 ' lr-AdamW/pg1‚0.00028815150583411494 ' lr-AdamW/pg2‚0.00028815150583411494 trainer/global_step‚899 # _timestamp‚1.78843827119438e+09+~ü*ÈÜ trainer/global_step‚899 # _timestamp‚1.78843827119438e+09 _runtime‚ 711.239023318 _step‚68 ' lr-AdamW/pg1‚0.00028815150583411494 ' lr-AdamW/pg2‚0.00028815150583411494DžÉ0Ý+ _wandbruntime‚711  _runtime‚711XlÉ‘– ÞÝO UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚77394944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚49260544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1645120240794 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16466127 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16466127 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8278785 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95740715774976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884382711951838e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98831742276608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12819951 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95740741202944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_poolw“½M.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚69 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2666264576 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1647718517914 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14367667 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16484663 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98829116442624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12819957 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29286078 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85480611143680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97385836015770 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12819957 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1650423477248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1647824034816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6089356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2276493536256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 711.240089307 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97388459720858 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98829116442624 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97385836015770 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12819957 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1650423477248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1647824034816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6089356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2276493536256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 711.240089307 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97388459720858 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98829116442624 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16466127 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16466127 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8278785 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95740715774976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884382711951838e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98831742276608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12819951 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95740741202944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚69 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2666264576 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1647718517914 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14367667 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16484663 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98829116442624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12819957 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29286078 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85480611143680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14383539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95847056949248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 trainer/global_step‚899 _step‚70 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6095558 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97289232113152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6096033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95847082377216 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1649656363520 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8287981 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 % _timestamp‚1.7884382719365807e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85575525338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16502923 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97289232113152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1649656363520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8287984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2278970417152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97289258504704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98938888476672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12834193 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12834187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97496630238006 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12834187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1652255819264 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚77381632 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97494006532710 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1646949583462 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98941514323968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97289258504704 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1649547860790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16484361 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16502923 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚ 711.981817549 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2279019664384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49247232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1652255819264 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98941514323968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272r¿_ŒâëM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 trainer/global_step‚899 _step‚70 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6095558 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97289232113152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6096033 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95847082377216 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1649656363520 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8287981 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 % _timestamp‚1.7884382719365807e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85575525338624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16502923 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97289232113152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1649656363520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8287984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2278970417152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocµ=(&eated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97289258504704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98938888476672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12834193 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12834187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97496630238006 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12834187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1652255819264 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚77381632 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97494006532710 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1646949583462 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98941514323968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97289258504704 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1649547860790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16484361 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16502923 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚ 711.981817549 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2279019664384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49247232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1652255819264 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98941514323968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12834193 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16484361 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14383539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95847056949248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074FX240ã+ _wandbruntime‚711  _runtime‚711 ®•×NäÈ _runtime‚ 711.983151253 .loss/FRt1/assignment_loss‚3.09183651511952 1loss/SRqqt3/detection_loss‚2.5138903420145917 /loss/FBt1/detection_loss‚1.8628812759455542 trainer/global_step‚899 .loss/FRt2/assignment_loss‚4.73882576412168 /loss/FRt4/assignment_loss‚4.066508604420556 2loss/SRqqt1/assignment_loss‚2.3707443249981797 1loss/SRqqt3/assignment_loss‚2.937850876552303 0loss/FBt2/assignment_loss‚0.5045361327938735 /loss/FRt1/detection_loss‚1.7823445937138493 /loss/FRt2/detection_loss‚2.2340420402329544 .loss/FRt3/detection_loss‚2.583624075759541 0loss/FBt1/assignment_loss‚0.2429285987068105 1loss/FBt3/assignment_loss‚0.48852328533561934 1loss/SRqqt1/detection_loss‚1.5132089645337703 _step‚71 1loss/SRqqt4/assignment_loss‚2.328395114644714 $loss/total_loss‚62.0310633395175 epoch‚0 % _timestamp‚1.7884382719385993e+09 1loss/FBt4/assignment_loss‚0.37886439707440683 .loss/FRt4/detection_loss‚2.785170460740725 1loss/SRqqt4/detection_loss‚2.1762080846921257 -loss/FBt3/detection_loss‚3.59886882855342 .loss/FRt3/assignment_loss‚6.65380410714583 1loss/SRqqt2/assignment_loss‚2.205629932338541 1loss/SRqqt2/detection_loss‚2.0048745659264653 .loss/FBt2/detection_loss‚4.995884537696838 /loss/FBt4/detection_loss‚3.9716179164556356H«Ù_RåÌ 1loss/SRqqt2/assignment_loss‚2.205629932338541 1loss/SRqqt2/detection_loss‚2.0048745659264653 .loss/FBt2/detection_loss‚4.995884537696838 /loss/FBt4/detection_loss‚3.9716179164556356 _runtime‚ 711.983151253 .loss/FRt1/assignment_loss‚3.09183651511952 1loss/SRqqt3/detection_loss‚2.5138903420145917 /loss/FBt1/detection_loss‚1.8628812759455542 trainer/global_step‚899 .loss/FRt2/assignment_loss‚4.73882576412168 /loss/FRt4/assignment_loss‚4.066508604420556 2loss/SRqqt1/assignment_loss‚2.3707443249981797 1loss/SRqqt3/assignment_loss‚2.937850876552303 0loss/FBt2/assignment_loss‚0.5045361327938735 /loss/FRt1/detection_loss‚1.7823445937138493 /loss/FRt2/detection_loss‚2.2340420402329544 .loss/FRt3/detection_loss‚2.583624075759541 0loss/FBt1/assignment_loss‚0.2429285987068105 1loss/FBt3/assignment_loss‚0.48852328533561934 1loss/SRqqt1/detection_loss‚1.5132089645337703 _step‚71 1loss/SRqqt4/assignment_loss‚2.328395114644714 $loss/total_loss‚62.0310633395175 epoch‚0 % _timestamp‚1.7884382719385993e+09 1loss/FBt4/assignment_loss‚0.37886439707440683 .loss/FRt4/detection_loss‚2.785170460740725 1loss/SRqqt4/detection_loss‚2.1762080846921257 -loss/FBt3/detection_loss‚3.59886882855342 .loss/FRt3/assignment_loss‚6.65380410714583GÀSp”æ‚(:‰ ‡ÎåÔßøëÈ gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñù™ç‚(:Œ ‡ÎåÔŒ•ÿË disk.rbd2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 908.921875 disk.nvme0c0n1.in‚ 11.734375! disk.dm-0.out‚3690.9091796875 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 30.21484375 disk.sda1.out‚0* proc.memory.percent‚17.632222175598145 disk.rbd0.out‚0 disk.rbd2.out‚40.89892578125 disk.dm-0.in‚ 11.734375$ disk.nvme0n1.out‚3690.9091796875 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.out‚ 30.21484375$ disk./.usageGB‚1100.814956665039 disk.dm-1.out‚ 30.21484375 disk.sda1.in‚0 disk.rbd3.out‚0 cpu‚25.038340669163425 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 11.734375 disk.sda3.in‚0# memory_percent‚94.4523811340332# proc.memory.rssMB‚ 2888.86328125 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 1.129117e+06 network.recv‚337664) disk./.usagePercent‚73.87661538706045& disk.nvme0c0n1.out‚3690.9091796875 disk.rbd1.in‚0=Ž~“è‚(:ˆ –ÎåÔÆ¡úÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚97.176' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4352 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÒ,é‚(:Œ –ÎåÔðà´Ë% disk./.usageGB‚1100.8129959106445 disk.rbd1.out‚ 0.306640625 disk.sda1.in‚0 disk.nvme0n1.in‚ 11.734375 disk.nvme0c0n1.in‚ 11.734375! disk.dm-0.out‚3691.5732421875 disk.rbd2.out‚42.00048828125 disk.dm-1.out‚ 30.8515625* proc.memory.percent‚17.632222175598145 network.sent‚ 1.178075e+06 disk.sda2.out‚0$ disk.nvme0n1.out‚3691.5732421875 disk.rbd0.out‚0 disk.dm-0.in‚ 11.734375 proc.cpu.threads‚17 network.recv‚342590 disk.sda.out‚ 30.8515625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.out‚ 30.8515625 disk.dm-1.in‚0$ memory_percent‚94.44949626922607# proc.memory.rssMB‚ 2888.86328125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 909.39453125 cpu‚25.03893618291872) disk./.usagePercent‚73.87648379918785 disk.rbd0.in‚0& disk.nvme0c0n1.out‚3691.5732421875r«2’ê‚(:‡ ¥ÎåÔó•äÈ gpu.0.gpu‚4.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚226.803' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.3606 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ço^ë‚(:Ž ¥ÎåÔª·ôË disk.sda.in‚0 disk.dm-0.in‚ 11.734375# proc.memory.rssMB‚ 2888.86328125 disk.rbd0.out‚0" disk.dm-0.out‚3790.92919921875 disk.drbd1777.out‚0 disk.sda.out‚ 31.7734375 disk.dm-1.out‚ 31.7734375# memory_percent‚94.4481372833252 network.sent‚ 1.184135e+06 disk.sda3.out‚ 31.7734375 disk.dm-1.in‚0* proc.memory.percent‚17.632222175598145 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚3790.92919921875 proc.cpu.threads‚17% disk./.usageGB‚1100.8149490356445 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 11.734375 network.recv‚346832) disk./.usagePercent‚73.87661487504539 disk.drbd1777.in‚0 disk.rbd2.out‚43.21533203125% disk.nvme0n1.out‚3790.92919921875 disk.sda3.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 11.734375' proc.memory.availableMB‚ 909.6171875 cpu‚25.038795377737983W§b†0ì+ _wandbruntime‚750  _runtime‚750ÁNÄí¾ _runtime‚ 750.553335582 _step‚72 & lr-AdamW/pg1‚0.0003041777297403505 & lr-AdamW/pg2‚0.0003041777297403505 trainer/global_step‚949 % _timestamp‚1.7884383105086846e+09½Êq[Èî & lr-AdamW/pg1‚0.0003041777297403505 & lr-AdamW/pg2‚0.0003041777297403505 trainer/global_step‚949 % _timestamp‚1.7884383105086846e+09 _runtime‚ 750.553335582 _step‚72Høºé=0ï+ _wandbruntime‚750  _runtime‚750ÉòìQî'ðèO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104320358392832 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102578277675008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15164584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104317732558848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6425999 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1739199242522 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚73 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15165061 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1742054326272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13531751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90226377317376 % _timestamp‚1.7884383105094073e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102578304066560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102794375453978 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6426473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8738588 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2666264576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90226302019584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1739454883840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102578277675008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚ 750.55427415 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17399715 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1739454883840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87823569543168 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13531757 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102578304066560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17399715 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17381179 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2402779639808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102796999159066 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104320358392832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30931472 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104317732558848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87823597677568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0M7«˜%ñìO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17399715 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1739454883840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87823569543168 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13531757 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102578304066560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17399715 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17381179 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚47163392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2402779639808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102796999159066 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104320358392832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30931472 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚75297792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104317732558848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87823597677568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104320358392832 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102578277675008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15164584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104317732558848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6425999 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1739199242522 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚73 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15165061 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1742054326272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13531751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90226377317376 % _timestamp‚1.7884383105094073e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102578304066560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102794375453978 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6426473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8738588 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2666264576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90226302019584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 A7DeviceStatsMonitor.on_train_batch_start/active.all.£ííèqpeak‚22903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1739454883840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102578277675008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚ 750.55427415 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646I9ÔïV0ò+ _wandbruntime‚751  _runtime‚751¬ËàHù&óóM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102686243771904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15180997 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90321192506880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2405282060288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2405234910208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13545993 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102902545970918 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13545987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104427504592896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13545987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15180554 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8747781 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90321267791360 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87915957596672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30945400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104430130440192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1741287212544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101164115662848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102686217380352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101164141090816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17399413 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8747784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1743886668288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17417975”ÄLý&ô÷M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17399413 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8747784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1743886668288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17417975 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102686243771904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15180997 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90321192506880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2405282060288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2405234910208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13545993 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102902545970918 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13545987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104427504592896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13545987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15180554 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8747781 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90321267791360 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87915957596672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30945400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104430130440192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1741287212544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101164115662848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102686217380352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101164141090816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376JCâñ“0õ+ _wandbruntime‚751  _runtime‚751džSöÍ 0loss/FRt1/assignment_loss‚3.1845498724681573 .loss/FRt3/assignment_loss‚6.50096997347745 _step‚75 /loss/FRt4/assignment_loss‚3.773281625338963 2loss/SRqqt3/assignment_loss‚2.9920986890792847 2loss/SRqqt4/assignment_loss‚1.9367279731759839 0loss/FBt2/assignment_loss‚0.4262366408136306 0loss/FBt4/assignment_loss‚0.6545842847091128 /loss/FRt3/detection_loss‚2.6136821616779673 .loss/FRt4/detection_loss‚2.261254119021552 /loss/FBt3/detection_loss‚3.4914666468446907 0loss/SRqqt1/assignment_loss‚2.21165953979629 %loss/total_loss‚57.32367489043253 /loss/FRt1/detection_loss‚2.3187381316248965 0loss/FBt3/assignment_loss‚0.6386972408979339 1loss/SRqqt1/detection_loss‚1.6159019861064974 1loss/SRqqt3/detection_loss‚2.9888617419279537 epoch‚0 1loss/FBt1/assignment_loss‚0.21981899668917093 1loss/SRqqt2/detection_loss‚1.5789389099976787 1loss/SRqqt4/detection_loss‚1.8597964641510272 /loss/FBt2/detection_loss‚2.9601181470430813 .loss/FBt4/detection_loss‚2.787579272138445 /loss/FRt2/detection_loss‚2.1637853112432266 /loss/FRt2/assignment_loss‚4.683726781798947 trainer/global_step‚949 % _timestamp‚1.7884383112490218e+09 _runtime‚ 751.293691565 0loss/SRqqt2/assignment_loss‚2.20346351968099 /loss/FBt1/detection_loss‚1.2577368607295973ìÐÍW÷Ñ /loss/FRt3/detection_loss‚2.6136821616779673 .loss/FRt4/detection_loss‚2.261254119021552 /loss/FBt3/detection_loss‚3.4914666468446907 0loss/SRqqt1/assignment_loss‚2.21165953979629 %loss/total_loss‚57.32367489043253 /loss/FRt1/detection_loss‚2.3187381316248965 0loss/FBt3/assignment_loss‚0.6386972408979339 1loss/SRqqt1/detection_loss‚1.6159019861064974 1loss/SRqqt3/detection_loss‚2.9888617419279537 epoch‚0 1loss/FBt1/assignment_loss‚0.21981899668917093 1loss/SRqqt2/detection_loss‚1.5789389099976787 1loss/SRqqt4/detection_loss‚1.8597964641510272 /loss/FBt2/detection_loss‚2.9601181470430813 .loss/FBt4/detection_loss‚2.787579272138445 /loss/FRt2/detection_loss‚2.1637853112432266 /loss/FRt2/assignment_loss‚4.683726781798947 trainer/global_step‚949 % _timestamp‚1.7884383112490218e+09 _runtime‚ 751.293691565 0loss/SRqqt2/assignment_loss‚2.20346351968099 /loss/FBt1/detection_loss‚1.2577368607295973 0loss/FRt1/assignment_loss‚3.1845498724681573 .loss/FRt3/assignment_loss‚6.50096997347745 _step‚75 /loss/FRt4/assignment_loss‚3.773281625338963 2loss/SRqqt3/assignment_loss‚2.9920986890792847 2loss/SRqqt4/assignment_loss‚1.9367279731759839 0loss/FBt2/assignment_loss‚0.4262366408136306 0loss/FBt4/assignment_loss‚0.6545842847091128K®M€ðŸø‚(:” ´ÎåÔ¾–îÈ gpu.0.gpu‚84.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚242.417' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.483399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò«¡Pù‚(:Œ ´ÎåÔâôÖÊ disk.rbd0.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚3890.71044921875 disk.dm-0.in‚ 11.734375 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0$ memory_percent‚94.44701671600342 cpu‚25.039403742560705 network.recv‚350487 disk.rbd1.in‚0 disk.sda.out‚ 32.234375 disk.sda2.out‚0* proc.memory.percent‚17.632222175598145 disk.sda3.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1100.8151626586914 disk.sda3.out‚ 32.234375 disk.nvme0c0n1.in‚ 11.734375" disk.dm-0.out‚3890.71044921875 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 11.734375( proc.memory.availableMB‚ 909.80078125# proc.memory.rssMB‚ 2888.86328125 network.sent‚ 1.188935e+06' disk.nvme0c0n1.out‚3890.71044921875 disk.dm-1.out‚ 32.234375 disk.rbd3.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚73.8766292114673 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚43.24658203125 disk.rbd3.in‚0>–ºŸú‚(:” ÃÎåÔí¤‰É gpu.0.gpu‚82.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚260.861' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.172200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Ú¤kû‚(:‘ ÃÎåÔ뛽Ë disk.rbd2.out‚44.46142578125 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚94.44854259490967( proc.memory.availableMB‚ 909.55078125 network.recv‚356490 disk.rbd2.in‚0 disk.sda.out‚ 32.87890625 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 32.87890625' disk.nvme0c0n1.out‚3989.99169921875 disk.dm-0.in‚ 11.734375 disk.dm-1.out‚ 32.87890625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd3.in‚0% disk./.usageGB‚1100.8152923583984% disk.nvme0n1.out‚3989.99169921875* proc.memory.percent‚17.632246017456055 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 11.734375" proc.memory.rssMB‚ 2888.8671875 cpu‚25.03923910353943 network.sent‚ 1.239068e+06 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.87663791572346 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 11.734375" disk.dm-0.out‚3989.99169921875 disk.drbd1777.in‚0ÛS'0ü+ _wandbruntime‚789  _runtime‚789·ì0ÆýÀ _step‚76 ' lr-AdamW/pg1‚0.00032020395364658604 ' lr-AdamW/pg2‚0.00032020395364658604 trainer/global_step‚999 % _timestamp‚1.7884383496262546e+09 _runtime‚ 789.670938544Ð8¿wÊþÄ ' lr-AdamW/pg2‚0.00032020395364658604 trainer/global_step‚999 % _timestamp‚1.7884383496262546e+09 _runtime‚ 789.670938544 _step‚76 ' lr-AdamW/pg1‚0.00032020395364658604LtN¬¸0ÿ+ _wandbruntime‚789  _runtime‚7890*“Ÿ¾€éO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 trainer/global_step‚999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9198385 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2528501818880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1828081690010 % _timestamp‚1.7884383496270301e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1831085732864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108205538597274 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15962916 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94971551432192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1833685175296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109808974509056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18314767 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1830679967130 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6765005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14243557 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2528551079424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14243557 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18314767 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9198388 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94971474037248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1833685175296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1831085732864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14243551 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109806348675072 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6765005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14243557 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2528551079424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14243557 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18314767 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9198388 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94971474037248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1833685175296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1831085732864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14243551 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109806348675072 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15962916 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94971551432192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1833685175296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109808974509056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320MèÒ¸0‚+ _wandbruntime‚790  _runtime‚790-01Íù&ƒóM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9207584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95066363622400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108083229039104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1829911032678 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108083229039104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108313709114422 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18314465 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1835517517312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15978792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1832509310006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32572252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14257787 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9207581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 trainer/global_step‚999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1835517517312 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95066441004032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6771669 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884383503657231e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49247232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109918746556416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6771211 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92535388406272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 _runtime‚ 790.410929134 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1832918061568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14257793 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14257787 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109916120709120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106481174376448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109918746556416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108083202647552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚77381632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531003350528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108311085409126 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106481199804416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320€dR„÷M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 _runtime‚ 790.410929134 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1832918061568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14257793 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14257787 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109916120709120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106481174376448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109918746556416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108083202647552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚77381632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531003350528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108311085409126 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106481199804416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9207584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95066363622400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108083229039104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1829911032678 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108083229039104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108313709114422 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18314465 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1835517517312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15978792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1832509310006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32572252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14257787 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9207581 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 trainer/global_step‚999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1835517517312 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95066441004032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6771669 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884383503657231e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚49247232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109918746556416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6771211 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92535388406272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2666264576N’ä¦Ô0…+ _wandbruntime‚790  _runtime‚790·‹|ËQ†Ë .loss/FRt2/detection_loss‚2.013563727339109 /loss/FRt4/detection_loss‚2.6830843129219155 .loss/FBt4/detection_loss‚4.331744943346296 1loss/SRqqt1/detection_loss‚1.5131859433604404 1loss/SRqqt3/detection_loss‚2.5873439601949744 0loss/SRqqt4/detection_loss‚2.649061206408909 .loss/FBt3/detection_loss‚3.696619846604087 0loss/FRt4/assignment_loss‚4.5088225572537155 1loss/SRqqt2/assignment_loss‚2.120665011347317 .loss/FRt3/detection_loss‚3.088899075984955 %loss/total_loss‚59.49496107930209 epoch‚0 1loss/FBt2/assignment_loss‚0.33719636988561513 0loss/FBt4/assignment_loss‚0.6097110829661168 1loss/SRqqt2/detection_loss‚1.7031277164084013 /loss/FBt1/detection_loss‚0.9458179170645075 /loss/FRt1/assignment_loss‚2.668733220100403 1loss/SRqqt1/assignment_loss‚1.949969775043428 0loss/FBt1/assignment_loss‚0.2677260694002415 /loss/FRt1/detection_loss‚2.2603457717597486 /loss/FRt2/assignment_loss‚4.185034249226252 /loss/FRt3/assignment_loss‚6.513478226131863 1loss/SRqqt3/assignment_loss‚2.948963323154965 2loss/SRqqt4/assignment_loss‚2.3804315303053176 .loss/FBt2/detection_loss‚3.219406241720373 trainer/global_step‚999 _runtime‚ 790.412502779 0loss/FBt3/assignment_loss‚0.3120290013731309 % _timestamp‚1.7884383503678572e+09 _step‚79dwÂU‡Ï 1loss/SRqqt3/detection_loss‚2.5873439601949744 0loss/SRqqt4/detection_loss‚2.649061206408909 .loss/FBt3/detection_loss‚3.696619846604087 0loss/FRt4/assignment_loss‚4.5088225572537155 1loss/SRqqt2/assignment_loss‚2.120665011347317 .loss/FRt3/detection_loss‚3.088899075984955 %loss/total_loss‚59.49496107930209 epoch‚0 1loss/FBt2/assignment_loss‚0.33719636988561513 0loss/FBt4/assignment_loss‚0.6097110829661168 1loss/SRqqt2/detection_loss‚1.7031277164084013 /loss/FBt1/detection_loss‚0.9458179170645075 /loss/FRt1/assignment_loss‚2.668733220100403 1loss/SRqqt1/assignment_loss‚1.949969775043428 0loss/FBt1/assignment_loss‚0.2677260694002415 /loss/FRt1/detection_loss‚2.2603457717597486 /loss/FRt2/assignment_loss‚4.185034249226252 /loss/FRt3/assignment_loss‚6.513478226131863 1loss/SRqqt3/assignment_loss‚2.948963323154965 2loss/SRqqt4/assignment_loss‚2.3804315303053176 .loss/FBt2/detection_loss‚3.219406241720373 trainer/global_step‚999 _runtime‚ 790.412502779 0loss/FBt3/assignment_loss‚0.3120290013731309 % _timestamp‚1.7884383503678572e+09 _step‚79 .loss/FRt2/detection_loss‚2.013563727339109 /loss/FRt4/detection_loss‚2.6830843129219155 .loss/FBt4/detection_loss‚4.331744943346296 1loss/SRqqt1/detection_loss‚1.5131859433604404OZ±xpŸˆ‚(:” ÒÎåÔƒ¼É gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.568' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.913599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ2‰‚(:’ ÒÎåÔÕ™Ì disk.sda.out‚ 33.55078125$ memory_percent‚94.44887638092041* proc.memory.percent‚17.632246017456055 network.sent‚ 1.244293e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 33.55078125 disk.sda2.in‚0 disk.rbd0.out‚0 network.recv‚360496 disk.sda1.in‚0 disk.nvme0n1.in‚ 11.734375% disk.nvme0n1.out‚3990.48291015625' disk.nvme0c0n1.out‚3990.48291015625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚45.56689453125 cpu‚25.039694630634262 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 11.734375) disk./.usagePercent‚73.87663791572346 disk.sda1.out‚0" disk.dm-0.out‚3990.48291015625 disk.rbd1.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 909.49609375% disk./.usageGB‚1100.8152923583984 disk.rbd3.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 11.734375" proc.memory.rssMB‚ 2888.8671875 proc.cpu.threads‚17 disk.sda3.out‚ 33.55078125 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0V¡v “Š‚(:ˆ áÎåÔþ…‹É gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.713 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V#“½‹‚(:Ž áÎåÔÛÖÌ disk.rbd3.in‚0% disk.nvme0n1.out‚4089.66650390625 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2888.8671875 disk.dm-0.in‚ 11.734375 disk.sda.out‚ 34.36328125 cpu‚25.039581623449916 network.recv‚366003' disk./.usagePercent‚73.876638171731" disk.dm-0.out‚4089.66650390625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 34.36328125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚4089.66650390625 disk.rbd3.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚46.71923828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 909.203125% disk./.usageGB‚1100.8152961730957 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 11.734375* proc.memory.percent‚17.632246017456055 proc.cpu.threads‚17 disk.dm-1.out‚ 34.36328125 disk.nvme0n1.in‚ 11.734375$ memory_percent‚94.45066452026367 network.sent‚ 1.294126e+06m´z”Œ‚(:‰ ðÎåÔÓä“É gpu.0.gpu‚66.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\í’‚(:Š ðÎåÔ«€Ë) disk./.usagePercent‚73.87663945176868 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 11.734375 disk.rbd2.out‚46.75048828125 network.sent‚ 1.299904e+06 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 11.734375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 34.796875 disk.rbd2.in‚0$ memory_percent‚94.44928169250488' proc.memory.availableMB‚ 909.4296875* proc.memory.percent‚17.632246017456055 network.recv‚370150% disk.nvme0n1.out‚4189.08935546875 disk.sda3.out‚ 34.796875 disk.dm-0.in‚ 11.734375" disk.dm-0.out‚4189.08935546875 disk.sda.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚4189.08935546875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2888.8671875 cpu‚25.039837075590967$ disk./.usageGB‚1100.815315246582 disk.rbd3.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 34.796875NW£0Ž+ _wandbruntime‚829  _runtime‚829…ÆÀ _runtime‚ 829.168257709 _step‚80 ' lr-AdamW/pg1‚0.00033623017755282154 ' lr-AdamW/pg2‚0.00033623017755282154 trainer/global_step‚1049 $ _timestamp‚1.788438389123576e+09;¡ZÊÄ ' lr-AdamW/pg2‚0.00033623017755282154 trainer/global_step‚1049 $ _timestamp‚1.788438389123576e+09 _runtime‚ 829.168257709 _step‚80 ' lr-AdamW/pg1‚0.00033623017755282154P‹ÉFº0‘+ _wandbruntime‚829  _runtime‚829%r Üð'’êO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113611454330394 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14955357 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97062374893568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19229819 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1919562414618 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1925316024320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14955351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7100268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115294964791296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97062403027968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1925316024320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 trainer/global_step‚1049 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚433 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19229819 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884383891243834e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115297590625280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7099835 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2654474659328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0·¼¯ô'“îO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34166634 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14955357 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115294964791296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97062403027968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1925316024320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 trainer/global_step‚1049 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚433 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚73200640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19229819 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884383891243834e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115297590625280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7099835 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2654474659328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113611454330394 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14955357 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97062374893568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19229819 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1919562414618 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1925316024320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14955351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7100268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14969593 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1927148366336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111798233090048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16773826 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97154762947072 _runtime‚ 829.909118471 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9667381 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19229517 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115407362672640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1927148366336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚âàÉÍÐ 2657010098176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16774249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14969587 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19248079 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111798258518016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99811846232576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚420 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113722248552630 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 _step‚82 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115407362672640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99811773045248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19248079 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19229517 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7106865 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1049 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2657055151104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115404736825344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚423 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328{ ñþ&–øM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1049 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2657055151104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115404736825344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚423 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚73187328 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14969593 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1927148366336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111798233090048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16773826 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97154762947072 _runtime‚ 829.909118471 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9667381 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19229517 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115407362672640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1927148366336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2657010098176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16774249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14969587 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19248079 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111798258518016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99811846232576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚420 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113722248552630 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 _step‚82 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115407362672640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99811773045248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19248079 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19229517 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7106865 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FÄŸ¾ & lr-AdamW/pg2‚0.0003522564014590571 trainer/global_step‚1099 $ _timestamp‚1.788438428375149e+09 _runtime‚ 868.419845544 _step‚84 & lr-AdamW/pg1‚0.0003522564014590571w·UÈ Â _runtime‚ 868.419845544 _step‚84 & lr-AdamW/pg1‚0.0003522564014590571 & lr-AdamW/pg2‚0.0003522564014590571 trainer/global_step‚1099 $ _timestamp‚1.788438428375149e+09T'¬LS0¡+ _wandbruntime‚868  _runtime‚868ɶƒ±#¢ðO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15667157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118769259868160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2016946873344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20144871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2016946873344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101681777568768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120783580907520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17554387 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120783580907520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2013641416346 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119019993768602 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118769259868160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118769233476608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119022617473690 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2014347430912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20126335 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15667157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20126335 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7436402 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104462363406336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104462290205696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35793486 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117008950629376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2011043139226 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120786206741504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10117985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.largU3yDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20126335 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15667157 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20126335 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7436402 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104462363406336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104462290205696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35793486 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117008950629376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2011043139226 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120786206741504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10117985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10117988 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7436837 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117008976057344 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120786206741504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101681805703168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2780512636928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118769233476608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15667157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118769259868160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2016946873344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20144871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2016946873344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101681777568768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚45066240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120783580907520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17554387 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120783580907520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2013641416346UYDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117115291803648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2018779215360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117115317231616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104557191465984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20163131 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17570213 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20144569 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2016179759616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2783070896640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2015470759222 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20144569 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2018779215360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10127184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118877173181952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120893352941568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118877199573504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15681393 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20163131 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120895978788864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7443032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 869.16446351 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119128164285542 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 $ _timestamp‚1.788438429119059e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35825956 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15681393 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17570631 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118877173181952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22626 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚415 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2599455744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119130787990838 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117115291803648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2018779215360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117115317231616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104557191465984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20163131 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17570213 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20144569 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2016179759616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2783070896640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2015470759222 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20144569 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚418 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2018779215360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10127184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118877173181952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120893352941568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118877199573504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15681393 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20163131 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120895978788864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7443032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Vö!Î_0§+ _wandbruntime‚869  _runtime‚869Bêã0¨È 1loss/SRqqt2/assignment_loss‚2.267624603489698 1loss/SRqqt3/assignment_loss‚3.169201475201231 .loss/FBt2/detection_loss‚4.861427426338196 $loss/total_loss‚59.7095521465138 0loss/FBt4/assignment_loss‚0.6104923946162065 /loss/FRt1/detection_loss‚1.9769169199362135 /loss/FRt2/detection_loss‚2.5133241224796214 0loss/FBt1/assignment_loss‚0.2815437120775647 _step‚87 /loss/FRt3/detection_loss‚3.2114849307320337 trainer/global_step‚1099 /loss/FBt4/detection_loss‚3.1576306646068892 /loss/FRt3/assignment_loss‚5.725360090082342 1loss/SRqqt4/assignment_loss‚2.498523485881311 0loss/FBt2/assignment_loss‚0.6040283162146807 0loss/SRqqt2/detection_loss‚1.858339548616086 0loss/SRqqt4/detection_loss‚1.930590663795118 .loss/FBt3/detection_loss‚4.286083774907248 1loss/SRqqt1/assignment_loss‚1.968995519766682 1loss/SRqqt3/detection_loss‚2.7108137607574463 /loss/FRt4/assignment_loss‚4.220195585367631 0loss/FBt3/assignment_loss‚0.3023462489114276 .loss/FRt4/detection_loss‚2.473332153899329 1loss/SRqqt1/detection_loss‚1.3527183138618344 /loss/FBt1/detection_loss‚0.8076632631650689 epoch‚0 _runtime‚ 869.165759291 0loss/FRt1/assignment_loss‚2.3515235990808723 .loss/FRt2/assignment_loss‚4.56939157272907 $ _tim-µB†estamp‚1.788438429121151e+09W¥hBR©Ì /loss/FRt2/detection_loss‚2.5133241224796214 0loss/FBt1/assignment_loss‚0.2815437120775647 _step‚87 /loss/FRt3/detection_loss‚3.2114849307320337 trainer/global_step‚1099 /loss/FBt4/detection_loss‚3.1576306646068892 /loss/FRt3/assignment_loss‚5.725360090082342 1loss/SRqqt4/assignment_loss‚2.498523485881311 0loss/FBt2/assignment_loss‚0.6040283162146807 0loss/SRqqt2/detection_loss‚1.858339548616086 0loss/SRqqt4/detection_loss‚1.930590663795118 .loss/FBt3/detection_loss‚4.286083774907248 1loss/SRqqt1/assignment_loss‚1.968995519766682 1loss/SRqqt3/detection_loss‚2.7108137607574463 /loss/FRt4/assignment_loss‚4.220195585367631 0loss/FBt3/assignment_loss‚0.3023462489114276 .loss/FRt4/detection_loss‚2.473332153899329 1loss/SRqqt1/detection_loss‚1.3527183138618344 /loss/FBt1/detection_loss‚0.8076632631650689 epoch‚0 _runtime‚ 869.165759291 0loss/FRt1/assignment_loss‚2.3515235990808723 .loss/FRt2/assignment_loss‚4.56939157272907 $ _timestamp‚1.788438429121151e+09 1loss/SRqqt2/assignment_loss‚2.267624603489698 1loss/SRqqt3/assignment_loss‚3.169201475201231 .loss/FBt2/detection_loss‚4.861427426338196 $loss/total_loss‚59.7095521465138 0loss/FBt4/assignment_loss‚0.6104923946162065 /loss/FRt1/detection_loss‚1.9769169199362135WŒÃÉ”ª‚(:‰ ÏåÔÄžÿÈ gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚267.599' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú²«‚(:‰ ÏåÔíßêË network.sent‚ 1.361375e+06 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2888.8671875 proc.cpu.threads‚17) disk./.usagePercent‚73.87660284269127 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0# memory_percent‚94.4493055343628( proc.memory.availableMB‚ 909.42578125 disk.rbd1.in‚0 disk.sda.out‚ 37.1796875! disk.dm-0.out‚4391.1513671875 disk.dm-1.out‚ 37.1796875$ disk./.usageGB‚1100.814769744873$ disk.nvme0n1.out‚4391.1513671875 disk.sda3.out‚ 37.1796875& disk.nvme0c0n1.out‚4391.1513671875 cpu‚25.040299150744527 disk.sda1.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 14.296875 network.recv‚384075 disk.rbd2.out‚ 49.2197265625 disk.nvme0c0n1.in‚ 14.296875 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 14.296875* proc.memory.percent‚17.632246017456055Úlù%”¬‚(:‰ ¬ÏåÔ½¹É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚247.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W‹/ ­‚(:‚ ¬ÏåÔÞÕªË disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.04048678982998% disk./.usageGB‚1100.8150901794434 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 14.61328125 disk.dm-1.out‚37.8125 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 14.61328125$ memory_percent‚94.44985389709473 proc.cpu.threads‚17) disk./.usagePercent‚73.87662434732415 disk.rbd0.in‚0# disk.nvme0n1.out‚4492.736328125 network.sent‚ 1.41161e+06 network.recv‚390304 disk.rbd1.out‚ 0.306640625 disk.rbd2.out‚ 50.3251953125 disk.rbd3.out‚0' proc.memory.availableMB‚ 909.3359375* proc.memory.percent‚17.632246017456055 disk.sda3.out‚37.8125 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda.out‚37.8125 disk.dm-0.out‚4492.736328125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 14.61328125" proc.memory.rssMB‚ 2888.8671875 disk.rbd0.out‚0% disk.nvme0c0n1.out‚4492.736328125 disk.rbd1.in‚0ÃðTjŸ®‚(:” »ÏåÔ™àäÈ gpu.0.gpu‚89.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚240.832' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.166399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÆân¯‚(:” »ÏåÔ¸ ÖÌ) disk./.usagePercent‚73.87665942035633% disk./.usageGB‚1100.8156127929688 disk.rbd2.out‚ 51.5400390625 disk.sda2.in‚0% disk.nvme0n1.out‚4592.43408203125 network.recv‚394832 disk.sda.out‚ 38.3203125 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 15.48828125 disk.dm-0.in‚ 15.48828125* proc.memory.percent‚17.632246017456055 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚94.44835186004639( proc.memory.availableMB‚ 909.58203125 network.sent‚ 1.417772e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 15.48828125 disk.drbd1777.in‚0' disk.nvme0c0n1.out‚4592.43408203125 disk.dm-1.out‚ 38.3203125 disk.sda1.out‚0" disk.dm-0.out‚4592.43408203125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2888.8671875 cpu‚25.040565361183557 disk.sda3.out‚ 38.3203125òVQ„0°+ _wandbruntime‚907  _runtime‚907ª>/‡Æ±À _step‚88 ' lr-AdamW/pg1‚0.00036828262536529264 ' lr-AdamW/pg2‚0.00036828262536529264 trainer/global_step‚1149 $ _timestamp‚1.788438467848282e+09 _runtime‚ 907.892930145ŽŒwÊ²Ä _step‚88 ' lr-AdamW/pg1‚0.00036828262536529264 ' lr-AdamW/pg2‚0.00036828262536529264 trainer/global_step‚1149 $ _timestamp‚1.788438467848282e+09 _runtime‚ 907.892930145X]KÚ?0³+ _wandbruntime‚907  _runtime‚907VÙ¼—ô'´îO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124428533206810 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 907.8940047 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20787574272 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚77394944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2906260049408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623705088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122326034770944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1149 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126272197023744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21059923 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16378957 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7773237 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124431156911898 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126272197023744 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16378951 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124166218743808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2108577722368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18351488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10577788 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109207517688320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884384678490193e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124166218743808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2102523863834 _step‚89 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18351022 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21041387 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106301208378368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16378957 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124166245135360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126274822857728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21041387 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10577785 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2906309309952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124166245135360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2105978279936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2102523863834 _step‚89 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18351022 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21041387 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106301208378368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16378957 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124166245135360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126274822857728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21041387 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10577785 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 346301440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2906309309952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124166245135360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245623020 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2105978279936 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122326034770944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1149 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126272197023744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21059923 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16378957 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7773237 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚124431156911898 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126272197023744 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16378951 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124166218743808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2108577722368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18351488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10577788 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109207517688320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424457685 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2625833984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884384678490193e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124166218743808Y²àû0¶+ _wandbruntime‚908  _runtime‚908ákès ·úM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109302414595072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1149 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37471376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124539327429046 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106393568297472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚47150080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 346301440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2107810608640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124274184840704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124274158449152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126381969057792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 _step‚90 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126384594905088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7779845 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21059621 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7780285 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2106951483830 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 % _timestamp‚1.7884384685906503e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2908771013120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122432350517248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598277328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16393187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109302339310592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16393193 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21078183 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21059621 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3246802944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3246802944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106393596431872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16393187 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126384594905088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124274158449152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124274184840704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124536703723750 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2110410064384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10586981 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18366826 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 908.635985035 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16393193 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10586984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2104353206502 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122432375945216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚75284480 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124274158449152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124274184840704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124536703723750 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2110410064384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10586981 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18366826 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 908.635985035 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16393193 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10586984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2104353206502 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122432375945216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126381969057792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 _step‚90 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126384594905088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7779845 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21059621 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7780285 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2106951483830 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 % _timestamp‚1.7884384685906503e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2908771013120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Zb"Ã0¹+ _wandbruntime‚908  _runtime‚908 ³qžXºÒ 0loss/FRt3/assignment_loss‚6.0585528165102005 2loss/SRqqt1/assignment_loss‚1.7687980721168446 0loss/SRqqt2/assignment_loss‚1.92329048267642 1loss/SRqqt3/assignment_loss‚2.525105838264738 1loss/FBt1/assignment_loss‚0.25807636367577674 0loss/FBt4/assignment_loss‚0.6557134971153573 1loss/SRqqt2/detection_loss‚1.8352734808061943 trainer/global_step‚1149 .loss/FBt2/detection_loss‚4.601763367652893 0loss/FBt2/assignment_loss‚0.3923614174127579 /loss/FRt1/detection_loss‚1.8082619158048479 /loss/FRt4/detection_loss‚2.5575571419163183 /loss/FRt3/detection_loss‚2.7879373654723167 &loss/total_loss‚58.664203500142335 epoch‚0 0loss/FRt1/assignment_loss‚2.8380057821198115 /loss/FRt4/assignment_loss‚4.572456747293472 0loss/FBt3/assignment_loss‚0.5963638848625124 /loss/FRt2/detection_loss‚2.5583615075974238 .loss/FBt4/detection_loss‚3.337144473567605 _runtime‚ 908.63710718 1loss/SRqqt3/detection_loss‚2.8520425111055374 0loss/FRt2/assignment_loss‚4.4000888495218184 1loss/SRqqt4/detection_loss‚1.9868128415942192 /loss/FBt1/detection_loss‚1.1776695061495166 /loss/FBt3/detection_loss‚3.6460914611816406 % _timestamp‚1.7884384685924973e+09 _step‚91 2loss/SRqqt4/assignment_loss‚2.2120672768354415 1loss/SRqqt1/detection_loss‚1.3144068988886746à¤jþ\»Ö 0loss/FRt2/assignment_loss‚4.4000888495218184 1loss/SRqqt4/detection_loss‚1.9868128415942192 /loss/FBt1/detection_loss‚1.1776695061495166 /loss/FBt3/detection_loss‚3.6460914611816406 % _timestamp‚1.7884384685924973e+09 _step‚91 2loss/SRqqt4/assignment_loss‚2.2120672768354415 1loss/SRqqt1/detection_loss‚1.3144068988886746 0loss/FRt3/assignment_loss‚6.0585528165102005 2loss/SRqqt1/assignment_loss‚1.7687980721168446 0loss/SRqqt2/assignment_loss‚1.92329048267642 1loss/SRqqt3/assignment_loss‚2.525105838264738 1loss/FBt1/assignment_loss‚0.25807636367577674 0loss/FBt4/assignment_loss‚0.6557134971153573 1loss/SRqqt2/detection_loss‚1.8352734808061943 trainer/global_step‚1149 .loss/FBt2/detection_loss‚4.601763367652893 0loss/FBt2/assignment_loss‚0.3923614174127579 /loss/FRt1/detection_loss‚1.8082619158048479 /loss/FRt4/detection_loss‚2.5575571419163183 /loss/FRt3/detection_loss‚2.7879373654723167 &loss/total_loss‚58.664203500142335 epoch‚0 0loss/FRt1/assignment_loss‚2.8380057821198115 /loss/FRt4/assignment_loss‚4.572456747293472 0loss/FBt3/assignment_loss‚0.5963638848625124 /loss/FRt2/detection_loss‚2.5583615075974238 .loss/FBt4/detection_loss‚3.337144473567605 _runtime‚ 908.63710718 1loss/SRqqt3/detection_loss‚2.8520425111055374[³ ž¼‚(:“ ÊÏåÔòƒ÷È gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.634' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.92680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞPÿ½‚(:” ÊÏåÔæìË network.sent‚ 1.422998e+06 disk.sda3.in‚0 disk.sda3.out‚ 38.7890625 disk.rbd0.out‚0" disk.dm-0.out‚4692.74658203125 disk.rbd2.in‚0 disk.sda.out‚ 38.7890625$ memory_percent‚94.44868564605713 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625% disk.nvme0n1.out‚4692.74658203125' disk.nvme0c0n1.out‚4692.74658203125 disk.dm-1.in‚0 disk.sda1.in‚0 cpu‚25.041054388988567 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 38.7890625 disk.rbd2.out‚ 52.6455078125 disk.sda.in‚0( proc.memory.availableMB‚ 909.52734375* proc.memory.percent‚17.632246017456055) disk./.usagePercent‚73.87666402849196% disk./.usageGB‚1100.8156814575195 disk.nvme0n1.in‚ 15.48828125 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.recv‚398628 disk.sda2.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 15.48828125 disk.dm-0.in‚ 15.48828125 disk.sda1.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2888.8671875–;ÿ®”¾‚(:‰ ÙÏåÔÜÝ«É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚240.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÝY¿‚(: ÙÏåÔ¿—’Ë disk.nvme0c0n1.in‚16.40625$ memory_percent‚94.44992542266846 disk.rbd2.out‚ 52.8017578125 disk.sda2.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚4693.17626953125 disk.sda1.out‚0 disk.dm-1.out‚ 39.35546875) disk./.usagePercent‚73.87667657286113% disk./.usageGB‚1100.8158683776855 disk.drbd1777.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 909.32421875 disk.rbd1.out‚ 0.306640625 disk.sda2.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2888.87109375 disk.rbd2.in‚0% disk.nvme0n1.out‚4693.17626953125 disk.sda3.out‚ 39.35546875 disk.dm-0.in‚16.40625 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚4693.17626953125 cpu‚25.041205491429512 proc.cpu.threads‚17 network.recv‚404200 disk.sda.in‚0 disk.sda.out‚ 39.35546875* proc.memory.percent‚17.632269859313965 network.sent‚ 1.472846e+06 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚16.40625€±˜tÀ‚(:’ èÏåÔŠìšÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚94.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.830000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[húÁ‚(:… èÏåÔšŠøÊ disk.sda3.out‚ 40.03515625 disk.rbd1.out‚ 0.306640625% disk./.usageGB‚1100.8158493041992 disk.sda1.out‚0 disk.rbd3.in‚0 network.sent‚ 1.478793e+06 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚16.40625 disk.dm-0.in‚16.40625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚16.40625( proc.memory.availableMB‚ 909.26953125 disk.rbd0.in‚0 disk.dm-1.out‚ 40.03515625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 53.8837890625 disk.sda.out‚ 40.03515625 disk.rbd3.out‚0# memory_percent‚94.4502592086792# proc.memory.rssMB‚ 2888.87109375) disk./.usagePercent‚73.87667529282346$ disk.nvme0c0n1.out‚ 4792.37890625 disk.dm-0.out‚ 4792.37890625 disk.sda2.out‚0* proc.memory.percent‚17.632269859313965 cpu‚25.041094976695152 network.recv‚408628 disk.rbd0.out‚0" disk.nvme0n1.out‚ 4792.378906251Ú›0Â+ _wandbruntime‚947  _runtime‚947òRÄþ & lr-AdamW/pg1‚0.0003843088492715282 & lr-AdamW/pg2‚0.0003843088492715282 trainer/global_step‚1199 $ _timestamp‚1.788438507133891e+09 _runtime‚ 947.178573404 _step‚92ìêº,ÈÄ & lr-AdamW/pg1‚0.0003843088492715282 & lr-AdamW/pg2‚0.0003843088492715282 trainer/global_step‚1199 $ _timestamp‚1.788438507133891e+09 _runtime‚ 947.178573404 _step‚92\tÄ^0Å+ _wandbruntime‚947  _runtime‚947K?#ö'ÆðO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19147944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113952399790592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110920582919168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127643093484544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131760813139968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129563230402560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19147455 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113952320298496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129837072645018 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚93 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17090751 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17090757 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8109870 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2196602865562 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17090757 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3031788737024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120‡¤ØI´ÇôO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129563204011008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21974975 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21956439 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21956439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20787574272 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2599442432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMon‡žF%itor.on_train_batch_start/active.all.freed‚39047190 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2625833984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598277120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22626 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3246802944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19147944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113952399790592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110920582919168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127643093484544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131760813139968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129563230402560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19147455 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113952320298496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129837072645018 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚93 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17090751 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2599442432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17090757 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8109870 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22626 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2196602865562 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17090757 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3031788737024 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2202040913408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114047213904896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17104993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2199441457664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111012970972672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19163873 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129945243161958 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11046781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19163410 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2199441457664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114047293383680 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 _runtime‚ 947.923295085 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131873211021312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3034294276608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129671143716352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17104987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17104987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131873211021312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21993235 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21974673 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127749409230848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8117089 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129671170107904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2198432208438 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 _step‚94 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2195833931110 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129671170107904 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 $ _timestamp‚1.788438507878109e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8116629 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20787574272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129671170107904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2198432208438 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2625847296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245623020 _step‚94 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2195833931110 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129671170107904 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 $ _timestamp‚1.788438507878109e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8116629 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424457685 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2625847296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2202040913408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114047213904896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17104993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2199441457664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111012970972672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19163873 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129945243161958 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11046781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623705296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19163410 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18562 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2199441457664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114047293383680 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 _runtime‚ 947.923295085 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131873211021312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3034294276608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129671143716352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17104987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17104987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20787574272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131873211021312 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2599455744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21993235^Ám‰0Ë+ _wandbruntime‚947  _runtime‚947C$çùRÌÌ /loss/FRt4/detection_loss‚2.2932217551225964 2loss/SRqqt2/assignment_loss‚2.1394790429621935 .loss/FRt1/detection_loss‚2.129005860917422 1loss/SRqqt2/detection_loss‚1.9677323391661048 .loss/FRt3/assignment_loss‚5.72238645060309 1loss/SRqqt1/detection_loss‚1.4415529752298006 0loss/SRqqt4/detection_loss‚2.138249097764492 % _timestamp‚1.7884385078798084e+09 _runtime‚ 947.924331899 0loss/FBt1/assignment_loss‚0.4213129642551447 /loss/FBt1/detection_loss‚0.7602160441382837 0loss/FBt4/assignment_loss‚0.9757961452007293 /loss/FRt2/detection_loss‚2.3064185742426804 .loss/FBt2/detection_loss‚4.926737070083618 .loss/FBt3/detection_loss‚2.799744665622711 .loss/FRt2/assignment_loss‚4.00915954068855 2loss/SRqqt1/assignment_loss‚1.7245366632465333 2loss/SRqqt4/assignment_loss‚1.9476016093045474 .loss/FRt3/detection_loss‚2.610733332305119 1loss/SRqqt3/detection_loss‚2.6286225857273227 -loss/FBt4/detection_loss‚2.75377242565155 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚2.8743371659395645 /loss/FRt4/assignment_loss‚4.264622360467911 2loss/SRqqt3/assignment_loss‚2.7201746932921873 0loss/FBt3/assignment_loss‚0.5244352661584344 &loss/total_loss‚56.530274327443806 epoch‚0 _step‚95 0loss/FBt2/assignment_loss‚0.4504256993532181= %VÍÐ .loss/FRt3/assignment_loss‚5.72238645060309 1loss/SRqqt1/detection_loss‚1.4415529752298006 0loss/SRqqt4/detection_loss‚2.138249097764492 % _timestamp‚1.7884385078798084e+09 _runtime‚ 947.924331899 0loss/FBt1/assignment_loss‚0.4213129642551447 /loss/FBt1/detection_loss‚0.7602160441382837 0loss/FBt4/assignment_loss‚0.9757961452007293 /loss/FRt2/detection_loss‚2.3064185742426804 .loss/FBt2/detection_loss‚4.926737070083618 .loss/FBt3/detection_loss‚2.799744665622711 .loss/FRt2/assignment_loss‚4.00915954068855 2loss/SRqqt1/assignment_loss‚1.7245366632465333 2loss/SRqqt4/assignment_loss‚1.9476016093045474 .loss/FRt3/detection_loss‚2.610733332305119 1loss/SRqqt3/detection_loss‚2.6286225857273227 -loss/FBt4/detection_loss‚2.75377242565155 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚2.8743371659395645 /loss/FRt4/assignment_loss‚4.264622360467911 2loss/SRqqt3/assignment_loss‚2.7201746932921873 0loss/FBt3/assignment_loss‚0.5244352661584344 &loss/total_loss‚56.530274327443806 epoch‚0 _step‚95 0loss/FBt2/assignment_loss‚0.4504256993532181 /loss/FRt4/detection_loss‚2.2932217551225964 2loss/SRqqt2/assignment_loss‚2.1394790429621935 .loss/FRt1/detection_loss‚2.129005860917422 1loss/SRqqt2/detection_loss‚1.9677323391661048_ßfàx΂(:‰ ÷ÏåÔïÈžÉ gpu.0.gpu‚91.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚42.0 gpu.0.powerWatts‚278.344' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6688 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0g¸¾f.uncorrectedMemoryErrors‚0ÖÞûÏ‚(:ƒ ÷ÏåÔ«Ä„Í disk.rbd1.in‚0 disk.sda.out‚ 40.5859375$ memory_percent‚94.89374160766602 disk.sda1.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2961.22265625) disk./.usagePercent‚73.87666018837895 disk.dm-1.out‚ 40.5859375 disk.rbd2.out‚ 55.0830078125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚16.40625& proc.memory.availableMB‚ 836.609375% disk./.usageGB‚1100.8156242370605 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda2.in‚0 disk.nvme0n1.in‚16.40625 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚413175) proc.memory.percent‚18.07386875152588# disk.nvme0n1.out‚4891.712890625 disk.sda3.out‚ 40.5859375 disk.rbd0.out‚0% disk.nvme0c0n1.out‚4891.712890625 cpu‚25.065438325447857 proc.cpu.threads‚17 network.sent‚ 1.484989e+06 disk.sda3.in‚0 disk.nvme0c0n1.in‚16.40625 disk.dm-0.out‚4891.712890625 disk.drbd1777.out‚0LŠpn’Ђ(:‡ †ÐåÔÄ´„É gpu.0.gpu‚6.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.958' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nà<¸Ñ‚(:ù †ÐåÔòœÍ disk.sda1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚16.40625 disk.drbd1777.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.077898025512695 disk.rbd3.in‚0 network.recv‚418070 disk.nvme0c0n1.in‚16.40625 disk.dm-1.out‚41.0625 disk.rbd2.in‚0 disk.sda.out‚41.0625 disk.sda2.out‚0% disk./.usageGB‚1100.8156547546387 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.132330523997783 disk.rbd1.in‚0 disk.rbd2.out‚ 56.1572265625 network.sent‚ 1.534127e+06) disk./.usagePercent‚73.87666223643922$ disk.nvme0c0n1.out‚ 4991.27734375 disk.sda1.out‚0 disk.sda3.out‚41.0625$ memory_percent‚94.89891529083252( proc.memory.availableMB‚ 835.76171875 disk.dm-0.out‚ 4991.27734375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 0.306640625 disk.nvme0n1.in‚16.40625" disk.nvme0n1.out‚ 4991.27734375 disk.sda3.in‚0" proc.memory.rssMB‚ 2961.8828125 disk.drbd1777.in‚0 disk.sda.in‚0ª°h¡“Ò‚(:ˆ •ÐåÔ½–™É gpu.0.gpu‚91.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚37.0 gpu.0.powerWatts‚89.393' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Çl Ó‚(:‚ •ÐåÔξÊË disk.drbd1777.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.07875633239746 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 56.3291015625 disk.sda.out‚ 41.64453125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚16.40625) disk./.usagePercent‚73.87667785289881" disk.nvme0n1.out‚ 4991.54296875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚16.40625 disk.dm-0.out‚ 4991.54296875 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 41.64453125' proc.memory.availableMB‚ 835.7734375 network.recv‚422441% disk./.usageGB‚1100.8158874511719" proc.memory.rssMB‚ 2962.0234375 disk.rbd1.in‚0 network.sent‚ 1.540036e+06 disk.drbd1777.out‚0 disk.dm-1.out‚ 41.64453125$ memory_percent‚94.89884376525879 disk.sda1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚16.40625$ disk.nvme0c0n1.out‚ 4991.54296875 disk.dm-1.in‚0 cpu‚25.20199220479311»¬?áœÔ‚(:‘ ¤ÐåÔ¶Ú˜É gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚35.0 gpu.0.powerWatts‚88.563' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.712600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ»© Õ‚(: ¤ÐåÔ¨ÚýË disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 42.1484375 disk.rbd0.out‚0 cpu‚25.265175876178297 network.recv‚426865 disk.drbd1777.in‚0 disk.sda.out‚ 42.1484375 disk.sda2.out‚0 disk.dm-0.in‚16.40625 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.out‚ 5090.93359375 disk.nvme0c0n1.in‚16.40625% disk./.usageGB‚1100.8158874511719 disk.drbd1777.out‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.out‚ 5090.93359375& proc.memory.availableMB‚ 835.703125 disk.dm-0.out‚ 5090.93359375 disk.rbd2.out‚ 57.4033203125 disk.nvme0n1.in‚16.40625 disk.dm-1.out‚ 42.1484375 network.sent‚ 1.545802e+06 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0$ memory_percent‚94.89927291870117* proc.memory.percent‚18.078970909118652 proc.cpu.threads‚17) disk./.usagePercent‚73.87667785289881 disk.rbd0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2962.05859375'ÎLrÖbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP/ßži×bd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JP.ëoØbj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JP“‹1qÙbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP5_¹tÚbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPßÅôDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP¤‚ÓwÜbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPK³+7oÝbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPÏŠpÞbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPßjlqßbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPä7&”màbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPÐÝ7ûtábo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP²!'tâbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPìØ8\eãb` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPºG0vkäbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPG§ú]tåbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP´GDáuæbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPu¡‹mmçbh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP<:~èby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPYÿzébu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP8n»sêbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPôšápëbk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JPŽ Rlyìbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÆeý}míbh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPo(¶sîbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPR'D“rïbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¶†€×mðbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPÍ ÆŽtñbo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPõ]òbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP|‰&lóbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP³.”Ldôb_ Gñuùbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPûXxúbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP]—å‰tûbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPë±ã7mübh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JP‚>÷;sýbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP=p™|þbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPÏ}îºqÿbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPÊež›w€br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPSï iybt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÄʼnt‚bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP‰ Þ qƒbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPs:#[o„bj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPû1&p…bk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPUE3Œv†bq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPœŽÁ l‡bg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP~Üàpˆbk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JP'<ãˆj‰be BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JP ¬êuuŠbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP ×dEt‹bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP¯ž´ÞcŒb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPaLàövbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP/ÃkŽbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPo£ósbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP¿`òrbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPn¬Íkt‘bo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPUs½Šg’bb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JPwáòy“bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP—ogœt”bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPÚ!dx•bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP¢ç¥Õ~–by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPÛÀ÷Lq—bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP@U o˜bj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JP ¨QAl™bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JP¥\áhšbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JP4Ø&–n›bi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPæ÷\ímœbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPçë¡lbg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPœ’¥8jžbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP˜2â†dŸb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPˆ'Áz¡bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPFçù_e¢b` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JP„ždn£bi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JP+…¿n¤bi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JP¿=5k¥bf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPѱki¦bd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPcNƒot§bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPpYsw¨br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPØŒìòh©bc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPyX½lqªbl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JP`;Hšu«bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPá­¾½r¬bm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP2?7Le­b` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPVfp®bk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPwìåuv¯bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP°dçùg°bb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP¨öDn±bi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPˆƒ§“k²bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPèAšzl³bg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP½<©žv´bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPù”„Šyµbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÂó‹Én¶bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP¿×ñäl·bg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPëçsók¸bf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JPîÝÀ¸s¹bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPêgaèwºbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP.3D¢oÀbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JP3.f*lÁbg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPºvZ×wÂbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPYtÃbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP6–'oÄbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP2°ÐôxÅbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPµWy’pÆbk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JPÒñesÇbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP•½šÔpÈbk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPÚnÂqÉbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP 9^ÉnÊbi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP䮽ƒ|Ëbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPþv¾µiÌbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPx/qyÍbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPPMXrÎbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPrkh vÏbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP5@p¸xÐbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPT4A™2Ñ- _wandbruntime‚1006  _runtime‚1006][_f`*ÒÚT \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1572 % _timestamp‚1.7884385660580454e+09 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚22495592 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3296722944 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1572 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚17766923 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20787574272 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚134256773968896 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245623020 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _step‚96 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2241782784512 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚117962185841152 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚134374633081854 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15193 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚134256798263296 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3246802944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚11518118 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚114881626955776 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23666360320 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1572 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚22495592 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2646 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚134372660951666 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23666360320 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2646 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚114881609307136 _runtime‚1006.103050938 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚40262510 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15188 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22626 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40277703 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950316032 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950316032 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚19912287 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 120444416 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2668361728 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2646 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚17766918 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1408 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚114 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 347481088 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚117962306285568 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23666360320 trainer/global_step‚1208 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚134256798263296 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚136498581047808 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22626 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚116 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8394169 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚22510780 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚136496606437376 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2238073915390 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15188 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 20369637376 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚134256773968896 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚40262510 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚136498581047808 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3246802944 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚11518116 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948799372 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1974610432 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8394055 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2646 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚136496606437376 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2239832468480 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3080679329792 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3080576534016 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972130188 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2241782784512 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424457685 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2239832468480 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20787574272 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17766923 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17766918 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚114881609307136 _runtime‚1006.103050938 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚40262510 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15188 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22626 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40277703 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950316032 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950316032 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚19912287 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 120444416 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2668361728 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2646 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚17766918½¶3‡¼# TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1408 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚114 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 347481088 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚117962306285568 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23666360320 trainer/global_step‚1208 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚134256798263296 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚136498581047808 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22626 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚116 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8394169 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚22510780 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚136496606437376 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2238073915390 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15188 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 20369637376 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚134256773968896 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚40262510 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚136498581047808 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3246802944 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚11518116 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948799372 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1974610432 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8394055 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2646 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚136496606437376 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2239832468480 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3080679329792 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3080576534016 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972130188 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2241782784512 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424457685 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2239832468480 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20787574272 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17766923 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17766918 Bf×ba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP·DîkØbf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JPª¥/ÞeÙb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPÛÓZnÚbi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPL±²lÛbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPÊÆNwÜbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPsgèJnÝbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPùcØWoÞbj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP"RËFjßbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP6Ôiàbd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPâÜÛ1zábu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP$Д>vâbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPù÷«vfãba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPòC—_säbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÖ$Õmåbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JP4î1Älæbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JP£ëD¢pçbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPÿ8³0rèbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPØX)Gjébe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPça#lêbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP?/FOnëbi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPƒ‰”kìbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP|ͱjíbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP½Éºtîbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPû|‚gïbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP—âuðbp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPõ5Ž„rñbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPÿWÿšvòbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPÄ2tóbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP+ÀOÎwôbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP’íÈ—qõbl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPA\:ìoöbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JP¾uo÷bj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPJÀà‡tøbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP̚܂uùbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPˆD7Dsúbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP…¬—çdûb_ Žc–b^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP—ÓŠxr—bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPäðúe˜b` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JPóOk™bf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPÂxî‡ršbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPâj›be BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP{Àýpœbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPUœD tbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP˜ÞÌš|žbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP‡5iwŸbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP®Ê³k bf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPÖiIñq¡bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPpÎw¢br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP8âr£bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP !°o¤bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPó¯Âs¥bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPIÖPTv¦bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP,ðdd§b_ ¯oÅbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JPš\¨ZmÆbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPÓpáZlÇbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JP*±y±uÈbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPð<†rÉbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP]˜­mÊbh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPßéËmËbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JPy¼sìrÌbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPøfëÒmÍbh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JP­•KEqÎbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPÓŒú2Ï- _wandbruntime‚1007  _runtime‚1007“¥M‚#ÐãR OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 _step‚97 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2242256987648 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948799372 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2236597801906 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788438567067591e+09 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17777817 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2242256987648 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17777812 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23666360320 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1572 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 G>DeviceStatsMonitor.on_validat± çion_batch_end/segment.all.current‚2646 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚134330532556800 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22626 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15188 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22502806 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23666360320 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚1007.112646326 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40295811 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚118024762843136 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚11526048 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672¡âym)ÑçR WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2240306671616 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8398118 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788438567067591e+09 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17777817 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2242256987648 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17777812 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23666360320 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1572 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2646 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚134330532556800 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22626 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15188 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22502806 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23666360320 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚1007.112646326 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40295811 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚118024762843136 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚11526048 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 _step‚97 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2242256987648 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948799372 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2236597801906 B+daHƒbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP PlH„bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÚ’$„N…bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP@ŽuèJ†bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPüå¼ïN‡bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPµvÂXNˆbI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPê#DJ‰bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPR¢À_;Šb6 jet/accuracy_1_of_7"trainer/global_step2:JP´ìâêJ‹bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŒ3à˜JŒbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP,¡mjJbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP÷nÆ JŽbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÔ‘vJbE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP(²×PJbE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP¸ö TN‘bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP~gecH’bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP›½%;“b6 jet/accuracy_7_of_7"trainer/global_step2:JPÀJ‘H”bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP}vvJ•bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÅ3>âH–bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPº};—b6 jet/accuracy_3_of_5"trainer/global_step2:JP`iæÒH˜bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPXa¨H™bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPˆÒJHšbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPeƒH›bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¼úº¦HœbC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP°oƒJbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP[P¿EJžbE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŸQis;Ÿb6 jet/accuracy_5_of_7"trainer/global_step2:JP%ˆÒÜH bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP£?ÝH¡bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPºeÛJ¢bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP=<±£H£bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPiöŒLJ¤bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP£º«‡N¥bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÖø‰¹J¦bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP=@•ŠN§bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP˜òó­H¨bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÙ…KN©bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPݦÙ)JªbE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP@Fw\N«bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP¢¶ ê@¬b; particle/accuracy_1_of_4"trainer/global_step2:JPÃnuóH­bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP ¢ëGH®bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPi´æÕH¯bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPS÷bH°bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPcéÈÌH±bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPUTÁ#H²bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP:nU¤H³bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP]ÜH´bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPš (!NµbI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPþX%ë;¶b6 jet/accuracy_4_of_5"trainer/global_step2:JPù‚/1H·bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPQœÄJ¸bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPh…¸H¹bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPtEJºbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPôá—J»bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPdOõÙN¼bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPaÐÔ>H½bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPvÜÙ˜H¾bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPô°îeJ¿bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPz @[JÀbE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP*§ç‰HÁbC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPGR CJÂbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPê鉕NÃbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP £qHÄbC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPdz°ØHÅbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPPkONÆbI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP#ÀrrJÇbE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPsƒG1HÈbC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP/¾ÛºHÉbC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÌARŽHÊbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPJö]GHËbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP×ËmJÌbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPäDE*HÍbC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPNÐݧJÎbE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPòêHÏbC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP‹Þ”HÐbC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP 6Ñ]HÑbC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÄ×TûHÒbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPJv¿7HÓbC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPãbNÔbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¿`2NÕbI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP‡ýôHÖbC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPϱ‡'H×bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPŽÅ³@Øb; particle/accuracy_3_of_7"trainer/global_step2:JPÂb6HÙbC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPQžéNÚbI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPù|XéHÝbC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPäU®JÞbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP7Y HßbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¼ø£pHàbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÔAÏNábI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÑN®NâbI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPèÖrNãbI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPcx2;äb6 jet/accuracy_1_of_1"trainer/global_step2:JPq´,ÅHåbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPf¸!cHæbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP/;qHçbC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPvõ:@NèbI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP ’Ö¾HébC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPˆí‚s@êb; particle/accuracy_5_of_7"trainer/global_step2:JPËa?JëbE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP:%°BNìbI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPxª¤zNíbI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPJ J©NîbI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP³VIHïbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP…4ØXNðbI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPØ>MwJñbE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP¹EÏHòbC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPsƒjrHóbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP kIÒNôbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPY¢î÷JõbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP$÷ìëNöbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP¼fdN÷bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPD !HøbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¾Ù6'HùbC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÎMIlNúbI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP¯â oNûbI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÁ›4 NübI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP ›‰£HýbC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPñV¡…NþbI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPU?âÙNÿbI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP[À,J€ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]KDc@ b; particle/accuracy_2_of_2"trainer/global_step2:JP»xJ‚ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP;Ž*Nƒ bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPüèÚJ„ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPéBÝJN… bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPà™=VN† bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPV„¤@‡ b; particle/accuracy_2_of_5"trainer/global_step2:JPMrK¹Jˆ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP$¡ÜÅJ‰ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÊz–NŠ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPë^q˜N‹ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP·&¬JŒ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP˜vÚüN bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP‘­:àNŽ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP ?_üN bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP}ƒ ë@ b; particle/accuracy_1_of_2"trainer/global_step2:JPúÝ £J‘ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPëËœ»J’ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP–‹êJ“ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¤–±N” bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP2rU;@• b; particle/accuracy_4_of_7"trainer/global_step2:JPM²ÑBN– bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPVXd¾N— bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP<{ PJ˜ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP6NµJ™ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPC­Q—Jš bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPìhN-N› bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPfÝqNœ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP F‹{H bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPg•*Nž bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP^¼¥tJŸ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPbš=N  bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPŸ%+H¡ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP€Þt:J¢ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPû½glN£ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP±â)YN¤ bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPPö'N¥ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPæ_ʺH¦ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP}ÝMN§ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP "õJ¨ bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPxì;© b6 jet/accuracy_2_of_5"trainer/global_step2:JPù5ëJª bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP]5˜N« bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP69 9N¬ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPöœúN­ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP3á[J® bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPw;bnN¯ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP.‰N° bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÏnjÍ;± b6 jet/accuracy_2_of_6"trainer/global_step2:JP)h9H² bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP»Y@H³ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPRÔJ´ bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‚:nNµ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÿpó·J¶ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP70^m;· b6 jet/accuracy_6_of_7"trainer/global_step2:JPøÿöàH¸ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPà3J¹ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPŸ÷aÇJº bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP–ðXJ» bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP'.ÔJ¼ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÂN0N½ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP$Ï-J¾ bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPSN ÂJ¿ bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPk¬äNÀ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPQ„²ÅHÁ bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPf¬R°H bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP=ê¸TNà bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP7_î¨NÄ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPKöõHÅ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPæéå8HÆ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP#@_XNÇ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP t8È b3 particle/f_score"trainer/global_step2:JPHÃ;É b6 jet/accuracy_3_of_7"trainer/global_step2:JPoZHÊ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPq"ÈHË bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP F­HÌ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP›—IdHÍ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP‚TîNÎ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP1×ÚsNÏ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP‰ÛÍjJÐ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPû' CJÑ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP/¶çaJÒ bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPÁTJÓ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPvý¾^NÔ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPµˆœJÕ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÔcØNÖ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP„J× bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPao@Ø b; particle/accuracy_2_of_6"trainer/global_step2:JP‡#·»JÙ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPÍüæLJÚ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPõQ»NÛ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPLùšNÜ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP¯›óDNÝ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¦kÆJÞ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP©IüVJß bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâµ³ÐJà bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÍõøJá bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÙ #Jâ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPb…ÀJã bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPèà2Jä bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPüŒæHå bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP X*Jæ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPá ÎNç bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP<è­ŒNè bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPê ðƒHé bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPSóä‘Nê bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÿS¥_Jë bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPTŒJì bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPuf´#Jí bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPfqwHî bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPæddêNï bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPP(ãNð bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP˜¾qONñ bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPC(|Hò bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÔ*‹Hó bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÁÁÝNô bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP„Æ–Nõ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÍ·äJö bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPùg?(N÷ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPökÎôJø bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPcHõHù bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÞᔞHú bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP,nPHû bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPUÁ®MJü bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP7qîNý bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÔžûHþ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶û ØHÿ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPËdJ€ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPñÀ¸J bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP-si—N‚ bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPT‚±°Jƒ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP`Æ?;„ b6 jet/accuracy_3_of_3"trainer/global_step2:JPÆè®RH… bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP?ü¤ÓJ† bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP#ê.H‡ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPlLoJˆ bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPó±¡îH‰ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¹õJŠ bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPIfy‰H‹ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP,K"HŒ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP”ѯêJ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPAéHŽ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPnÉC§N bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP)kã†H bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP–Ì9{H‘ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPUqXˆH’ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPMFH“ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP*W­‡J” bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÆÀ‰H• bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPŸx-J– bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÎŒœ¹H— bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPMΑäN˜ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÎX4'J™ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPìW&–Hš bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPŠÃƒ÷H› bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPko$5;œ b6 jet/accuracy_1_of_5"trainer/global_step2:JP1`CH bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPˆ’_@ž b; particle/accuracy_2_of_7"trainer/global_step2:JP<ÅCJŸ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÚ>AJ  bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPM‰v:N¡ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPŽñú;¢ b6 jet/accuracy_2_of_3"trainer/global_step2:JP÷‰¥"@£ b; particle/accuracy_1_of_3"trainer/global_step2:JPÍ9¬ÏH¤ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPï³¾ÕJ¥ bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPQÆ¡uJ¦ bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP´}&VN§ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP,uËN¨ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¶¬€^N© bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP8Ê Nª bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP£Ï]G« bB validation_average_jet_accuracy"trainer/global_step2:JPùÜe²J¬ bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‹ÌŠH­ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPµ8€(H® bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPâ²€J¯ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP›Ï…¬N° bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPô¦ ~;± b6 jet/accuracy_1_of_4"trainer/global_step2:JPO%MiJ² bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPdšÒ¢N³ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP‰ SH´ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP—¸:*Hµ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPµ˜Á‘J¶ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPv)Þ N· bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPFñÁ^N¸ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP”Ü­+J¹ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPò`A Nº bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP³j¦H» bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP÷óBN¼ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP#¸Õ@½ b; particle/accuracy_2_of_4"trainer/global_step2:JPEû*èN¾ bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPs(ù”H¿ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP µžJÀ bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¿NÁ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP3¼ÉQH bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP~ÝaNà bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPíö/§HÄ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP•vBNÅ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPbKNÆ bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPعóûHÇ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPlznP@È b; particle/accuracy_4_of_4"trainer/global_step2:JPòaÓHÉ bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPZã£)JÊ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÛòRNË bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPRå=NÌ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP2«;¶NÍ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP&XNÎ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¨ß+NÏ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPøÔ‘šJÐ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP5©]@Ñ b; particle/accuracy_7_of_7"trainer/global_step2:JPIå^JÒ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¼ÂœNÓ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP š:œJÔ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP°HÕ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPœþ&NÖ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP$ßOH× bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP­øF¼JØ bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPEvZàJÙ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPˆñвNÚ bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÎ漫JÛ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP©qŠHÜ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP©ó…BNÝ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP¿u1ÇNÞ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP W³Hß bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÐT¤²Jà bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP³ iHá bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP®œ†¤Nâ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP Y_æNã bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP÷n&Jä bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPg …Jå bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPªPúœNæ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÙ€ÕNç bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP8iŠÂHè bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP¼äøHé bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPKË_Hê bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP/a5Në bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÝÑG¯Hì bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPUp‹Hí bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPQ¸ÿJî bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPò¨“ëJï bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP“>ÒoHð bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¦<ÏJñ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP´,uHò bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÐò¿šJó bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP&Þ%Nô bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP§HóìHõ bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP™ä2 Jö bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP2E1N÷ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP8§¾nNø bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP ¿°¹Nù bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP[ýõ@ú b; particle/accuracy_1_of_1"trainer/global_step2:JP—×ßJû bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP9õJü bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPyÙ÷GHý bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP™Ÿ’­Hþ bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPZwHÿ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP«ØVÎ;€ b6 jet/accuracy_6_of_6"trainer/global_step2:JP„¹àúJ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP¯)vFJ‚ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP8sKNÞ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¸•;$Hß bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP´Z>Jà bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP샃ô<á b7 particle/specificity"trainer/global_step2:JPÁʬHâ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPSÙHã bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¾v…+Jä bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÓ¸äâHå bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPûi‹9æ b4 particle/accuracy"trainer/global_step2:JP{(w”Hç bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP_"Õ8Nè bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP˜‰„=Né bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP;ÅNê bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPà259Hë bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPs”UHì bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPVsÄNí bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPdûî„Hî bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¥,FdNï bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÌ/˜fNð bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÔ¦Jñ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP`X¾ÒHò bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP¥bmHó bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP´{ KJô bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!1n^Jõ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPzÕˆ,@ö b; particle/accuracy_5_of_6"trainer/global_step2:JPêá£H÷ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPfÌê?;ø b6 loss/val_total_loss"trainer/global_step2:JP‚X*ƒHù bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPùl&Hú bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP«ºHû bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP«‚ë^Nü bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPY)aJý bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP UƒÛJþ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP?ž“ñJÿ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPmjD‚N€ bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP.{pKJ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÏϱzN‚ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPãx¨TJƒ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÕ¼J„ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP£‘5´J… bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP }JÔ@† b; particle/accuracy_3_of_6"trainer/global_step2:JPâ¡×gJ‡ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPøÎù¨Nˆ bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPã¢C¶N‰ bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP•®–êJŠ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP4e-H‹ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÐ#¦JŒ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPDhKJJ bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¹7´üHŽ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP¢s$J bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPJM64H bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPWI³J‘ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÃ3káJ’ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPSZ'øJ“ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP][™H” bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP:Ú}H• bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP6×ÛH– bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÑÈ×îN— bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPþ…¨N˜ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPKn¯¸J™ bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÙ°ÅàHš bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP_Ê)H› bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPFÞó6Jœ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPh7kºN bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÎêÛNž bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÅ„õHŸ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPé?ÃC@  b; particle/accuracy_1_of_7"trainer/global_step2:JP ¯*H¡ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP³W"ÿJ¢ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP•yÈØH£ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPTì"CJ¤ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÁM‡²J¥ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPvÝ~ªJ¦ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPuzà‹J§ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPϺßÖJ¨ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP;V¶J© bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPEº^§Nª bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÝŒÖN« bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP2MexJ¬ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÝEŠ}N­ bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÑÞ^N® bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPóÓ¦Š@¯ b; particle/accuracy_1_of_5"trainer/global_step2:JP¶ÎbÏ;° b6 jet/accuracy_5_of_6"trainer/global_step2:JPnM ÀN± bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP6=kJ² bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP6HN³ bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP´h—*N´ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPå‡NJµ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP*ÖÆÖ;¶ b6 jet/accuracy_3_of_6"trainer/global_step2:JP þB J· bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP·æ|LJ¸ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP‡°útJ¹ bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÓ=$Jº bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPmJ» bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPª&SJ¼ bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÏÇ5N½ bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÓ=õàH¾ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP AZßN¿ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP@õª$HÀ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPê8CXNÁ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP˜hRJ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPs'g Jà bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÖ.W\JÄ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÑNÅ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPö #;Æ b6 jet/accuracy_2_of_7"trainer/global_step2:JP)ûzJÇ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPJQnJÈ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÜ[cJÉ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPq®åJÊ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPFŠË²HË bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPéãjJÌ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP|Ð2éJÍ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPŠ¯®jNÎ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPˆ‚§úHÏ bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP6Šf‡JÐ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPsÞJÑ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP§ L!JÒ bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÚ¬`JÓ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP]søJÔ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPïÕ0NÕ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPEÎnJÖ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPµ»AyN× bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPõ§NØ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPä‡2NÙ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP; JÚ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP"3 oHÛ bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP:éDNÜ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP<Ö£JÝ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPôg¸yHÞ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP®¢zFNß bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPS 7Jà bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPþìâJá bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPúL†9Nâ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPº—®Nã bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÏÀO}Hä bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP4=“xHå bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPb5ôJæ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¦YxÝHç bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÜÔ1Jè bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPNÌÓÞHé bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP|%pÊJê bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPRR9ANë bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÒ“¦Jì bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¡y’ÎJí bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÂ“Ä Nî bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP£ü@ï b; particle/accuracy_4_of_5"trainer/global_step2:JP†Ðà³Hð bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPgïzHñ bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP–?Í6;ò b6 jet/accuracy_4_of_7"trainer/global_step2:JPˆªsWJó bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP¢‚¦MNô bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPz_SEJõ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP(5pÊJö bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPUìjN÷ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP&„)vJø bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP2%;Nù bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP,*ZHú bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP@U$¶Jû bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP–gvŸNü bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPnüÁ´Ný bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPX»]˜Nþ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP°îü:Jÿ bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPEÌ‘°H€ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP èJ bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPUŠ)èN‚ bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÍ©ØÆ@ƒ b; particle/accuracy_3_of_5"trainer/global_step2:JPé ÷iJ„ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPƒõØÌJ… bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP RñyJ† bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPDªN‡ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP]j¡Nˆ bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÃÙíËN‰ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¼¿ÌõJŠ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¥½J‹ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP"ä¹HŒ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPƒ#ç5H bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP䚘cJŽ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP)²M~H bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPJˆ‘²H bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPÌ?ž{H‘ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP§ÊïAH’ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPÌ XwH“ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPŠ²LìN” bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÉŒ’J• bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPWiªFN– bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP ПA;— b6 jet/accuracy_1_of_3"trainer/global_step2:JPVÍH˜ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPa»&@J™ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP½Nš bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÕ9ÝWJ› bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP|ftlJœ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¢¯½J bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPµ˜^Nž bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÒ>ÃHŸ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP^˜çJ  bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPrÒˆ¸H¡ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÚ2»_J¢ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP{Ãê2£ - _wandbruntime‚1020  _runtime‚1020â̆\h¤ è¦ =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.008490566164255142 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9863086938858032 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.36063215136528015 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 jet/accuracy_4_of_7‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11408776044845581 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3995782136917114 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.34212011098861694 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4224780797958374 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878056645393372 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06940541416406631 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.35090509057044983 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666615009308 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.18421052396297455 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0529349111020565 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7325697541236877 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2811785042285919 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02298850566148758 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2805010676383972 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3125000298023224 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08539094030857086 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2776980996131897 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14735843241214752 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23869721591472626 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9840828776359558 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20842862129211426 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01344086043536663 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *jet/accuracy_1_of_2‚0.6226823329925537 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3413967192173004 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9705901741981506 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.09166666865348816 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3329935371875763 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.31361088156700134 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.36079299449920654 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08899126946926117 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3139877915382385 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.32614943385124207 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 *jet/accuracy_1_of_6‚0.9230769276618958 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014735664241015911 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0033316758926957846 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.29714909195899963 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 jet/accuracy_1_of_7‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37048688530921936 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.34152233600616455 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3809523582458496 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2517164349555969 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _runtime‚1020.760787953 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28727397322654724 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +jet/accuracy_3_of_5‚0.08645673096179962 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166666269302368 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.31481149792671204 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9784225225448608 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9840057492256165 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.032346922904253006 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_5_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15211918950080872 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14887455105781555 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036166951060295105 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2836257517337799 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10174163430929184 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9864195585250854 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.18580244481563568 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9653608798980713 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1772817075252533 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.21590909361839294 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3229166567325592 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4004462659358978 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.1944444477558136 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12545885145664215 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3800871968269348 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.11574073880910873 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1282377541065216 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10211537033319473 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.09090909361839294 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.30263552069664 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841017723083496 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0758577212691307 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 )jet/accuracy_1_of_1‚0.391220360994339 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2315972000360489 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.956250011920929 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.10536038875579834 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11965339630842209 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.20588235557079315 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12255594879388809 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.111111119389534 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.29618921875953674 particle/accuracy_2_of_2‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_2_of_5‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3887915313243866 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.28772398829460144 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34600383043289185 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 epoch‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10863717645406723 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09213406592607498 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14148734509944916 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6333601474761963 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32243114709854126 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978957772254944 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 *jet/accuracy_2_of_5‚0.3716782033443451 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.36538460850715637 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014430014416575432 jet/accuracy_6_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17309875786304474 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978957772254944 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02083333395421505 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8130923509597778 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009700176306068897 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 (particle/f_score‚0.33530017733573914 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.27906978130340576 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3844780921936035 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3125000298023224 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.27984747290611267 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3906656503677368 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2910958230495453 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11718323081731796 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07666119933128357 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3008597791194916 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07769589126110077 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9960809350013733 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2705244719982147 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38555291295051575 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08734572678804398 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.352083295583725 ,jet/accuracy_3_of_3‚0.015164666809141636 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.03333333507180214 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07875512540340424 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06447073072195053 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.0839184895157814 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9895035028457642 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3382372260093689 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.08445946127176285 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18083079159259796 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12415952980518341 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983012080192566 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008354219025932252 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.05330938100814819 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4478687047958374 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.06948523223400116 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07911238819360733 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.24676890671253204 *jet/accuracy_1_of_5‚0.8727726340293884 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4111247956752777 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 +jet/accuracy_2_of_3‚0.18559467792510986 particle/accuracy_1_of_3‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03787878900766373 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2847222089767456 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.35540080070495605 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9841271042823792 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.30232638120651245 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.7948206067085266 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010100402869284153 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.17931033670902252 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03576080501079559 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9770151972770691 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0820881575345993 particle/accuracy_4_of_4‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 _step‚98 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08481986075639725 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.40013861656188965 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4016663134098053 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9991645812988281 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2520800530910492 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2834291458129883 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2547961175441742 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08161980658769608 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218749403953552 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2118055373430252 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999701976776 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03348550945520401 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29226192831993103 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.20648150146007538 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 jet/accuracy_6_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.27178245782852173 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 *jet/accuracy_2_of_4‚0.2651815116405487 -jet/accuracy_4_of_4‚0.0017691287212073803 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00578379537910223 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt*FöeZ‚+Bt/event_proportion‚0.00018601190822664648 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26696425676345825 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0828065276145935 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39343610405921936 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3298946022987366 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246033430099487 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_1_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0026970701292157173 particle/accuracy_5_of_5‚1 +particle/sensitivity‚0.2672094702720642 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.20525527000427246 jet/accuracy_5_of_5‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_3_of_4‚1 particle/accuracy_6_of_7‚1 jet/accuracy_4_of_6‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.30773547291755676 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994982123374939 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.34212011098861694 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29242727160453796 particle/accuracy_3_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19609075784683228 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2696544826030731 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1785714328289032 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.31661897897720337 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03743743151426315 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚1208 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07666119933128357 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9397799968719482 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07719948887825012 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.23868772387504578 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08141762018203735 (particle/accuracy‚0.8347541093826294 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9929378032684326 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.06608187407255173 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.30685797333717346 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.21875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3478718101978302 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 )loss/val_total_loss‚56.56954955757447 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889941215515137 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10222862660884857 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26592200994491577 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4074286222457886 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04126845300197601 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4304570257663727 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ,"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.375 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26647067070007324 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13184551894664764 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.35081422328948975 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9852136373519897 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.32436424493789673 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_7‚1 % _timestamp‚1.7884385807139711e+09 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9821280837059021 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013321511447429657 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.28614071011543274 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6631931662559509 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2959531247615814 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_5‚1 jet/accuracy_5_of_6‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3888888955116272 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.35226333141326904 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9934707880020142 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.228451207280159 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 jet/accuracy_2_of_7‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4388953447341919 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.042553190141916275 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0035714285913854837 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10041618347167969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15185515582561493 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3218894302845001 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0476190485060215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98246103525161744)wßT¥ ì¦ 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.29714909195899963 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 jet/accuracy_1_of_7‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37048688530921936 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.34152233600616455 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3809523582458496 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2517164349555969 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _runtime‚1020.760787953 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28727397322654724 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +jet/accuracy_3_of_5‚0.08645673096179962 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166666269302368 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.31481149792671204 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9784225225448608 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9840057492256165 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.032346922904253006 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_5_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15211918950080872 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14887455105781555 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036166951060295105 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2836257517337799 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10174163430929184 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9864195585250854 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.18580244481563568 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9653608798980713 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1772817075252533 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.21590909361839294 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3229166567325592 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4004462659358978 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.1944444477558136 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12545885145664215 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3800871968269348 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.11574073880910873 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1282377541065216 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10211537033319473 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.09090909361839294 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.30263552069664 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841017723083496 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0758577212691307 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 )jet/accuracy_1_of_1‚0.391220360994339 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2315972000360489 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.956250011920929 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.10536038875579834 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11965339630842209 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.20588235557079315 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12255594879388809 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.111111119389534 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.29618921875953674 particle/accuracy_2_of_2‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_2_of_5‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3887915313243866 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.28772398829460144 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34600383043289185 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 epoch‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10863717645406723 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09213406592607498 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.14148734509944916 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6333601474761963 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32243114709854126 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978957772254944 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 *jet/accuracy_2_of_5‚0.3716782033443451 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.36538460850715637 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014430014416575432 jet/accuracy_6_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17309875786304474 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978957772254944 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02083333395421505 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8130923509597778 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009700176306068897 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 (particle/f_score‚0.33530017733573914 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.27906978130340576 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3844780921936035 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3125000298023224 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.27984747290611267 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3906656503677368 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2910958230495453 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11718323081731796 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07666119933128357 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3008597791194916 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07769589126110077 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9960809350013733 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2705244719982147 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38555291295051575 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08734572678804398 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.352083295583725 ,jet/accuracy_3_of_3‚0.015164666809141636 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.03333333507180214 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07875512540340424 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06447073072195053 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.0839184895157814 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9895035028457642 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3382372260093689 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.08445946127176285 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18083079159259796 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12415952980518341 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983012080192566 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008354219025932252 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.05330938100814819 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4478687047958374 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.06948523223400116 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07911238819360733 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.24676890671253204 *jet/accuracy_1_of_5‚0.8727726340293884 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4111247956752777 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 +jet/accuracy_2_of_3‚0.18559467792510986 particle/accuracy_1_of_3‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03787878900766373 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2847222089767456 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.35540080070495605 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9841271042823792 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.30232638120651245 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.7948206067085266 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010100402869284153 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.17931033670902252 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03576080501079559 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9770151972770691 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0820881575345993 particle/accuracy_4_of_4‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 _step‚98 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08481986075639725 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.40013861656188965 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4016663134098053 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9991645812988281 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2520800530910492 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2834291458129883 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2547961175441742 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08161980658769608 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218749403953552 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2118055373430252 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999701976776 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03348550945520401 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29226192831993103 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.20648150146007538 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 jet/accuracy_6_of_6‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.27178245782852173 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 *jet/accuracy_2_of_4‚0.2651815116405487 -jet/accuracy_4_of_4‚0.0017691287212073803 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00578379537910223 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26696425676345825 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0828065276145935 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.39343610405921936 =&Purity/3FRt2SRqqt*FBt/Ú=£”>event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3298946022987366 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246033430099487 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_1_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0026970701292157173 particle/accuracy_5_of_5‚1 +particle/sensitivity‚0.2672094702720642 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.20525527000427246 jet/accuracy_5_of_5‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_3_of_4‚1 particle/accuracy_6_of_7‚1 jet/accuracy_4_of_6‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.30773547291755676 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.994982123374939 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.34212011098861694 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29242727160453796 particle/accuracy_3_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19609075784683228 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2696544826030731 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1785714328289032 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.31661897897720337 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03743743151426315 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚1208 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07666119933128357 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9397799968719482 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07719948887825012 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.23868772387504578 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08141762018203735 (particle/accuracy‚0.8347541093826294 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9929378032684326 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.06608187407255173 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.30685797333717346 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.21875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3478718101978302 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 )loss/val_total_loss‚56.56954955757447 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889941215515137 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10222862660884857 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.26592200994491577 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4074286222457886 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04126845300197601 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4304570257663727 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ,"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.375 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26647067070007324 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13184551894664764 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.35081422328948975 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9852136373519897 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.32436424493789673 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_7‚1 % _timestamp‚1.7884385807139711e+09 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9821280837059021 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.013321511447429657 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.28614071011543274 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6631931662559509 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2959531247615814 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_5‚1 jet/accuracy_5_of_6‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3888888955116272 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.35226333141326904 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9934707880020142 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.228451207280159 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 jet/accuracy_2_of_7‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4388953447341919 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.042553190141916275 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0035714285913854837 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10041618347167969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.15185515582561493 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3218894302845001 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0476190485060215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9824610352516174 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.008490566164255142 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9863086938858032 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.36063215136528015 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 jet/accuracy_4_of_7‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11408776044845581 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3995782136917114 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.34212011098861694 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4224780797958374 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878056645393372 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06940541416406631 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.35090509057044983 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2866666615009308 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.18421052396297455 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0529349111020565 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7325697541236877 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2811785042285919 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02298850566148758 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2805010676383972 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3125000298023224 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08539094030857086 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2776980996131897 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14735843241214752 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23869721591472626 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9840828776359558 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20842862129211426 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01344086043536663 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *jet/accuracy_1_of_2‚0.6226823329925537 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3413967192173004 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9705901741981506 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.09166666865348816 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3329935371875763 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.31361088156700134 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.36079299449920654 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08899126946926117 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3139877915382385 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.32614943385124207 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 *jet/accuracy_1_of_6‚0.9230769276618958 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014735664241015911 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0033316758926957846by¸÷Ìž¦ ‚(:“ ÂÐåÔ÷ê™Ì gpu.0.gpu‚78.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.772000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ‹“$§ ‚(:™ ÂÐåÔêôŽÍ disk.dm-1.in‚0 disk.sda.out‚ 43.39453125 network.recv‚436673 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1700.49609375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1700.24609375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.845202445983887 cpu‚25.30329995641685% disk./.usageGB‚1104.4674377441406 disk.rbd3.in‚0# proc.memory.rssMB‚ 2759.91796875 network.sent‚ 1.622225e+06( disk./.usagePercent‚74.1217364569147 disk.dm-0.in‚ 1700.24609375 disk.rbd1.out‚ 0.306640625 disk.rbd2.in‚0 disk.sda3.out‚ 43.39453125 disk.dm-1.out‚ 43.39453125 disk.sda2.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚12593.927734375 disk.rbd1.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 3633.6328125 disk.sda1.out‚0$ disk.nvme0n1.out‚12593.927734375& disk.nvme0c0n1.out‚12593.927734375 disk.rbd2.out‚ 58.6181640625$ memory_percent‚77.82206535339355 proc.cpu.threads‚17rs…¿”¨ ‚(:‰ ÑÐåÔ”–É gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.807' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hz#© ‚(:˜ ÑÐåÔϲ®Ë$ memory_percent‚77.88498401641846 disk.rbd0.in‚0& disk.nvme0c0n1.out‚12594.189453125! disk.dm-0.out‚12594.189453125 disk.sda2.out‚0) disk./.usagePercent‚74.12175053732909 disk.rbd1.in‚0$ disk.nvme0n1.out‚12594.189453125 disk.sda3.in‚0# proc.memory.rssMB‚ 2760.49609375 disk.dm-0.in‚ 1718.67578125 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1718.67578125 disk.dm-1.in‚0) proc.memory.availableMB‚ 3623.32421875 proc.cpu.threads‚17 network.sent‚ 1.745449e+06% disk./.usageGB‚1104.4676475524902 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.306640625! disk.nvme0n1.in‚ 1718.67578125 disk.sda3.out‚ 44.0078125 cpu‚25.299626221287486 disk.dm-1.out‚ 44.0078125 disk.rbd2.out‚ 59.8330078125 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.84873104095459 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 44.0078125 disk.rbd3.in‚0 network.recv‚442225 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda1.out‚0ê×iû2ª - _wandbruntime‚1064  _runtime‚1064 +ãÈ«  trainer/global_step‚1249 % _timestamp‚1.7884386247753775e+09 _runtime‚1064.820109184 _step‚99 ' lr-AdamW/pg1‚0.00038750117279496985 ' lr-AdamW/pg2‚0.00038750117279496985<’éy̬ Æ ' lr-AdamW/pg1‚0.00038750117279496985 ' lr-AdamW/pg2‚0.00038750117279496985 trainer/global_step‚1249 % _timestamp‚1.7884386247753775e+09 _runtime‚1064.820109184 _step‚99cá2­ - _wandbruntime‚1064  _runtime‚1064’tïø'® òO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41836096 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18488873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3189342517760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3189296120320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139607002605568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119444917616640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚1064.821133514 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11996971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141926107609088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8717494 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119444889482240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18488879 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23327185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319131395072 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23347217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139606976214016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141928734209024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20714894 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20714462 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8717923 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141926107609088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18488873 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319131395072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139607002605568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122634260134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139688101461687 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2317879913143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141928734209024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11996968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23327185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139685477749059 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137370196120576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2315281628483 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968ýüiê¶ ¯ öO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139606976214016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46397440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 347481088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18488873 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3189342517760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3189296120320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139607002605568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119444917616640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 _runtime‚1064.821133514 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11996971 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141926107609088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8717494 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119444889482240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18488879 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 YJDeviceStatsM¹gáßFonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23327185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319131395072 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23347217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139606976214016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141928734209024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20714894 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20714462 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8717923 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141926107609088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18488873 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319131395072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139607002605568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122634260134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139688101461687 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2317879913143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141928734209024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11996968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23327185 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139685477749059 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137370196120576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2315281628483 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41836096 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122729115629568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18503109 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122729188050944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12006164 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72421376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚101 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142038506256384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚412 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23345419 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20730281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139796271978835 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884386255187676e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23345419 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142035879643136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8724526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8724117 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20730693 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139793648265999 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137476562722816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139714942310912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139714942310912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2323563945472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2319709256019 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23365477 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚409 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119537277535744 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139714915919360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2323563945472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142038506256384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18503115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12006167 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2320963723776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320963723776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944›ò`ê'´ €N trainer/global_step‚1249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20730693 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139793648265999 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137476562722816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139714942310912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139714942310912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2323563945472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2319709256019 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23365477 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚409 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119537277535744 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139714915919360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2323563945472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142038506256384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18503115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12006167 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2320963723776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320963723776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139714915919360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3191882380800 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122729115629568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18503109 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122729188050944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12006164 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72421376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚101 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142038506256384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚412 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23345419 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20730281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139796271978835 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884386255187676e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23345419 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142035879643136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8724526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8724117 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0e¢¬;Ê2µ - _wandbruntime‚1065  _runtime‚1065Õã1úT¶ Î /loss/FBt3/assignment_loss‚0.620087533910604 .loss/FBt2/detection_loss‚3.637240481376648 % _timestamp‚1.7884386255204172e+09 2loss/SRqqt2/assignment_loss‚2.1391974004178214 0loss/FBt1/assignment_loss‚0.2776819912513241 0loss/FBt4/assignment_loss‚0.5667766685087722 /loss/FRt4/detection_loss‚2.5448701292276383 1loss/SRqqt1/detection_loss‚1.1988351543034825 epoch‚1 1loss/SRqqt3/detection_loss‚2.5285836781774247 0loss/SRqqt4/detection_loss‚2.002647278904915 .loss/FBt3/detection_loss‚3.104917572094844 2loss/SRqqt4/assignment_loss‚1.8680599804222584 /loss/FBt1/detection_loss‚0.8915195990241221 .loss/FBt4/detection_loss‚2.781762203344932 #loss/total_loss‚54.270048362767 _step‚102 /loss/FRt1/assignment_loss‚2.846265184879303 /loss/FRt2/assignment_loss‚3.991553446079822 /loss/FRt3/assignment_loss‚5.768854190563333 2loss/SRqqt3/assignment_loss‚2.3984740938459126 /loss/FRt1/detection_loss‚2.1721255987882615 /loss/FRt2/detection_loss‚2.4126364810669676 _runtime‚1065.565039365 1loss/SRqqt1/assignment_loss‚1.793766234602247 .loss/FRt3/detection_loss‚2.913939219096611 trainer/global_step‚1249 0loss/FRt4/assignment_loss‚3.9477340251207353 1loss/FBt2/assignment_loss‚0.08202040079049766 1loss/SRqqt2/detection_loss‚1.7804998169685233C™ŒX· Ò .loss/FBt3/detection_loss‚3.104917572094844 2loss/SRqqt4/assignment_loss‚1.8680599804222584 /loss/FBt1/detection_loss‚0.8915195990241221 .loss/FBt4/detection_loss‚2.781762203344932 #loss/total_loss‚54.270048362767 _step‚102 /loss/FRt1/assignment_loss‚2.846265184879303 /loss/FRt2/assignment_loss‚3.991553446079822 /loss/FRt3/assignment_loss‚5.768854190563333 2loss/SRqqt3/assignment_loss‚2.3984740938459126 /loss/FRt1/detection_loss‚2.1721255987882615 /loss/FRt2/detection_loss‚2.4126364810669676 _runtime‚1065.565039365 1loss/SRqqt1/assignment_loss‚1.793766234602247 .loss/FRt3/detection_loss‚2.913939219096611 trainer/global_step‚1249 0loss/FRt4/assignment_loss‚3.9477340251207353 1loss/FBt2/assignment_loss‚0.08202040079049766 1loss/SRqqt2/detection_loss‚1.7804998169685233 /loss/FBt3/assignment_loss‚0.620087533910604 .loss/FBt2/detection_loss‚3.637240481376648 % _timestamp‚1.7884386255204172e+09 2loss/SRqqt2/assignment_loss‚2.1391974004178214 0loss/FBt1/assignment_loss‚0.2776819912513241 0loss/FBt4/assignment_loss‚0.5667766685087722 /loss/FRt4/detection_loss‚2.5448701292276383 1loss/SRqqt1/detection_loss‚1.1988351543034825 epoch‚1 1loss/SRqqt3/detection_loss‚2.5285836781774247 0loss/SRqqt4/detection_loss‚2.002647278904915fZ‚e”¸ ‚(:‰ ïÐåÔÀåöÉ gpu.0.gpu‚57.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚260.382' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^QU/¹ ‚(:“ ïÐåÔ¹Æ‚Ë network.sent‚ 1.849078e+06' proc.memory.availableMB‚ 3620.421875 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 62.0283203125 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚45.28125# memory_percent‚77.9026985168457 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.out‚0) proc.memory.percent‚16.85171127319336! disk.nvme0n1.in‚ 1718.67578125 disk.sda3.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12793.15478515625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1718.67578125 disk.dm-0.in‚ 1718.67578125 network.recv‚453093) disk./.usagePercent‚74.12164455020992% disk./.usage`9”¤GB‚1104.4660682678223 disk.sda2.in‚0 disk.sda.out‚45.28125 disk.sda1.in‚0 disk.sda3.out‚45.28125 cpu‚25.291973704844395 disk.rbd2.in‚0& disk.nvme0n1.out‚12793.15478515625# disk.dm-0.out‚12793.15478515625! proc.memory.rssMB‚ 2760.984375“?“Gžº ‚(:“ þÐåÔƨ’Ë gpu.0.gpu‚82.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚253.034' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.60679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÍå» ‚(:Š þÐåÔè¨Ì disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 cpu‚25.2886772769548 network.sent‚ 1.946331e+06 network.recv‚458435 disk.rbd1.in‚0 disk.sda.out‚ 45.9140625# memory_percent‚77.9062271118164! proc.memory.rssMB‚ 2760.984375 proc.cpu.threads‚17) disk./.usagePercent‚74.12164480621746 disk.rbd2.out‚ 62.0595703125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1718.67578125 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 12892.9296875& proc.memory.availableMB‚ 3619.84375 disk.dm-0.in‚ 1718.67578125 disk.sda1.out‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1718.67578125% disk./.usageGB‚1104.4660720825195 disk.drbd1777.out‚0) proc.memory.percent‚16.85171127319336 disk.dm-1.out‚ 45.9140625 disk.rbd1.out‚ 0.306640625 disk.sda2.in‚0" disk.nvme0n1.out‚ 12892.9296875 disk.sda3.out‚ 45.9140625 disk.rbd0.out‚0 disk.dm-0.out‚ 12892.9296875 disk.dm-1.in‚0ÈIi@2¼ - _wandbruntime‚1104  _runtime‚1104ô „6ɽ à _step‚103 ' lr-AdamW/pg1‚0.00038744868313877285 ' lr-AdamW/pg2‚0.00038744868313877285 trainer/global_step‚1299 % _timestamp‚1.7884386639554904e+09 _runtime‚1104.000161087ÁÅgJ; Ç % _timestamp‚1.7884386639554904e+09 _runtime‚1104.000161087 _step‚103 ' lr-AdamW/pg1‚0.00038744868313877285 ' lr-AdamW/pg2‚0.00038744868313877285 trainer/global_step‚1299g¼>ð 2¿ - _wandbruntime‚1104  _runtime‚1104œý²÷'À ñO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21511417 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124064292157440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2409360637751 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2410762244096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43462948 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24262269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147414723725312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19200679 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21510962 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3315368200704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2413362452480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145003987872768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24242237 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145094017187267 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9054194 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 $ _timestamp‚1.788438663956223e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2410762244096 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 trainer/global_step‚1299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147417350325248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19200673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚243996 L‰û'Á õO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2413362452480 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 trainer/global_step‚1299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147417350325248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19200673 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21511417 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124064292157440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2409360637751 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2410762244096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43462948 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24262269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147414723725312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19200679 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21510962 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3315368200704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2413362452480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145003987872768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24242237 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145094017187267 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9054194 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 $ _timestamp‚1.788438663956223e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2410762244096 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145111901186560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2415194794496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚1299 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21526963 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 1104.75816071 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19214915 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142793621436416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 # _timestamp‚1.78843866471244e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19214915 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚105 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9060999 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127474659482624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145204811417043 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2412594572800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2412594572800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2411189980627 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12465967 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24260471 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147527122372608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12465964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24280529 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145111901186560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3317951137280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43475380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2415194794496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2415194794496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145111901186560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2415194794496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚1299 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21526963 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 1104.75816071 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19214915 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142793621436416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 # _timestamp‚1.78843866471244e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19214915 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚105 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9060999 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127474659482624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145204811417043 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2412594572800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2412594572800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2411189980627 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12465967 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24260471 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147527122372608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12465964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24280529 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145111901186560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3317951137280i4 2Å - _wandbruntime‚1104  _runtime‚1104T“SÆ Í epoch‚1 0loss/FBt2/assignment_loss‚0.4973484984865146 .loss/FRt4/detection_loss‚2.137702030716119 1loss/SRqqt1/detection_loss‚1.3958176052341094 1loss/SRqqt4/detection_loss‚1.7974347819884617 /loss/FBt1/detection_loss‚1.1957423024354377 2loss/SRqqt4/assignment_loss‚1.9854087815792472 ,loss/FRt1/detection_loss‚2.1860709787876 /loss/FRt2/detection_loss‚2.3134792331530125 %loss/total_loss‚53.61861041911601 % _timestamp‚1.7884386647152667e+09 _runtime‚1104.759905098 /loss/FRt2/assignment_loss‚4.329041336263929 _step‚106 0loss/FBt1/assignment_loss‚0.2650440367209769 /loss/FBt2/detection_loss‚3.4609356267111644 .loss/FBt4/detection_loss‚2.251622033390132 trainer/global_step‚1299 /loss/FRt3/assignment_loss‚5.749881524306077 1loss/SRqqt3/assignment_loss‚2.884019210420806 0loss/FBt4/assignment_loss‚0.7106662417169322 /loss/FBt3/detection_loss‚2.7693663984537125 0loss/FRt1/assignment_loss‚2.3221054801753924 1loss/SRqqt2/assignment_loss‚1.709017582085668 0loss/FBt3/assignment_loss‚0.7764220098033547 2loss/SRqqt1/assignment_loss‚1.5500826248755821 .loss/FRt3/detection_loss‚2.620013420398419 1loss/SRqqt3/detection_loss‚2.6197175085544586 /loss/FRt4/assignment_loss‚4.007132013638814 0loss/SRqqt2/detection_loss‚2.084539159220092@cΦWÇ Ñ 1loss/SRqqt3/assignment_loss‚2.884019210420806 0loss/FBt4/assignment_loss‚0.7106662417169322 /loss/FBt3/detection_loss‚2.7693663984537125 0loss/FRt1/assignment_loss‚2.3221054801753924 1loss/SRqqt2/assignment_loss‚1.709017582085668 0loss/FBt3/assignment_loss‚0.7764220098033547 2loss/SRqqt1/assignment_loss‚1.5500826248755821 .loss/FRt3/detection_loss‚2.620013420398419 1loss/SRqqt3/detection_loss‚2.6197175085544586 /loss/FRt4/assignment_loss‚4.007132013638814 0loss/SRqqt2/detection_loss‚2.084539159220092 epoch‚1 0loss/FBt2/assignment_loss‚0.4973484984865146 .loss/FRt4/detection_loss‚2.137702030716119 1loss/SRqqt1/detection_loss‚1.3958176052341094 1loss/SRqqt4/detection_loss‚1.7974347819884617 /loss/FBt1/detection_loss‚1.1957423024354377 2loss/SRqqt4/assignment_loss‚1.9854087815792472 ,loss/FRt1/detection_loss‚2.1860709787876 /loss/FRt2/detection_loss‚2.3134792331530125 %loss/total_loss‚53.61861041911601 % _timestamp‚1.7884386647152667e+09 _runtime‚1104.759905098 /loss/FRt2/assignment_loss‚4.329041336263929 _step‚106 0loss/FBt1/assignment_loss‚0.2650440367209769 /loss/FBt2/detection_loss‚3.4609356267111644 .loss/FBt4/detection_loss‚2.251622033390132 trainer/global_step‚1299 /loss/FRt3/assignment_loss‚5.749881524306077j‡x±3”È ‚(:‰ ÑåÔ§âôÈ gpu.0.gpu‚84.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚192.893' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.5786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'u¡*É ‚(:Ÿ ÑåÔž¿Ë disk.sda.in‚0 disk.sda.out‚ 46.57421875 disk.sda2.out‚0 cpu‚25.28580855964995! disk.nvme0n1.in‚ 1718.67578125 disk.rbd0.out‚0# disk.dm-0.out‚12992.10009765625 disk.dm-1.out‚ 46.57421875 network.recv‚462768& disk.nvme0n1.out‚12992.10009765625 disk.rbd2.out‚63.22021484375 disk.sda1.out‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 3619.06640625) proc.memory.percent‚16.85171127319336 proc.cpu.threads‚17 disk.sda3.out‚ 46.57421875( disk.nvme0c0n1.out‚12992.10009765625 disk.sda2.in‚0$ memory_percent‚77.91097164154053% disk./.usageGB‚1104.4660720825195 disk.dm-1.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.12164480621746 disk.dm-0.in‚ 1718.67578125 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1718.67578125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.455078125 disk.sda1.in‚0! proc.memory.rssMB‚ 2760.984375 network.sent‚ 1.952306e+06 disk.drbd1777.in‚0 disk.rbd2.in‚0·€ìŠ”Ê ‚(:‰ œÑåÔ‹¯²É gpu.0.gpu‚68.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚216.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.3466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ™èÝ+Ë ‚(:  œÑåÔ£ £Ë network.sent‚ 2.060707e+06 disk.dm-0.in‚ 1718.67578125 disk.drbd1777.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚12992.59619140625 cpu‚25.282508793842034) disk./.usagePercent‚74.12164480621746) proc.memory.availableMB‚ 3619.01953125 disk.rbd1.out‚ 0.455078125 disk.sda.out‚ 47.06640625 disk.sda2.out‚0 disk.sda3.out‚ 47.06640625 proc.cpu.threads‚17# disk.dm-0.out‚12992.59619140625 disk.dm-1.in‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2760.984375) proc.memory.percent‚16.85171127319336% disk./.usageGB‚1104.4660720825195 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚64.25927734375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1718.67578125 disk.dm-1.out‚ 47.06640625 disk.drbd1777.out‚0$ memory_percent‚77.91125774383545 network.recv‚469266( disk.nvme0c0n1.out‚12992.59619140625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1718.67578125˜†¢‰”Ì ‚(:‰ «ÑåÔ§Þ³É gpu.0.gpu‚69.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â<¬½$Í ‚(:™ «ÑåÔ±­“Ë network.recv‚473750# disk.nvme0c0n1.in‚ 1718.67578125 disk.dm-0.in‚ 1718.67578125 disk.dm-1.in‚0 disk.dm-1.out‚ 47.8359375 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.4873046875 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2760.984375) disk./.usagePercent‚74.12165299845854 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚13091.67822265625 disk.rbd2.in‚0 disk.sda.out‚ 47.8359375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.066928e+06$ disk./.usageGB‚1104.466194152832# disk.dm-0.out‚13091.67822265625& disk.nvme0n1.out‚13091.67822265625$ memory_percent‚77.91376113891602 disk.drbd1777.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1718.67578125' proc.memory.availableMB‚ 3618.609375) proc.memory.percent‚16.85171127319336 cpu‚25.279525602405666 disk.rbd2.out‚ 64.412109375 disk.sda3.out‚ 47.8359375ÑúÀñ2Î - _wandbruntime‚1143  _runtime‚1143bg šÇÏ Á & lr-AdamW/pg1‚0.0003873558284359219 & lr-AdamW/pg2‚0.0003873558284359219 trainer/global_step‚1349 % _timestamp‚1.7884387034614294e+09 _runtime‚1143.506069644 _step‚107!ã:]ËÐ Å % _timestamp‚1.7884387034614294e+09 _runtime‚1143.506069644 _step‚107 & lr-AdamW/pg1‚0.0003873558284359219 & lr-AdamW/pg2‚0.0003873558284359219 trainer/global_step‚1349kIŠòÒ2Ñ - _wandbruntime‚1143  _runtime‚1143^¬†Æø'Ò òO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19912479 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150505180338103 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150502556625475 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚108 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150400973139968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25157289 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152903339841536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128683722967040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132125327243264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1143.506993926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19912473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22308133 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150400973139968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25157289 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22308598 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12916568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152903339841536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2504993301504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9392027 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 347702272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128683694832640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9391565 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2498243077699 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148004338975744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148004313547776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25177321 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132125403872256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152905966441472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552+÷TÓ öO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150400946748416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2502393093120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 347702272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128683694832640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9391565 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2498243077699 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148004338975744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148004313547776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25177321 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132125403872256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152905966441472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2o¯„¨626599936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19912479 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150505180338103 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150502556625475 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚108 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150400973139968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25157289 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152903339841536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128683722967040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132125327243264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1143.506993926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19912473 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22308133 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150400973139968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25157289 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22308598 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12916568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152903339841536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2504993301504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9392027 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2504225421824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22324454 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22323994 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 % _timestamp‚1.7884387041978185e+09 _step‚109 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45122296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153013111875584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150508886453760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128776111020544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153015738488832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25195581 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25175523 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2500072420367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148110680150016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25175523 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19926709 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572lˆš'Ö €N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19926709 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9398230 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132220224940032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506825643520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150613350855251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506825643520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2504225421824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2504225421824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22324454 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22323994 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 % _timestamp‚1.7884387041978185e+09 _step‚109 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45122296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153013111875584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150508886453760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128776111020544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153015738488832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25195581 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25175523 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2500072420367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148110680150016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25175523 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680m‘Wà82× - _wandbruntime‚1144  _runtime‚1144¥,ÚtYØ Ó /loss/FBt3/detection_loss‚3.5255911469459535 0loss/SRqqt1/detection_loss‚1.352770004980266 /loss/FBt1/detection_loss‚0.8338085578755078 /loss/FBt4/detection_loss‚3.8034308661114085 _runtime‚1144.244494034 /loss/FRt3/assignment_loss‚5.722967103123665 2loss/SRqqt2/assignment_loss‚2.1103066384792326 /loss/FRt1/detection_loss‚1.9763471196095148 /loss/FRt4/detection_loss‚2.3349740842114324 % _timestamp‚1.7884387041995735e+09 _step‚110 2loss/SRqqt3/assignment_loss‚2.3326998241245747 1loss/FBt2/assignment_loss‚0.12831966451065252 .loss/FRt2/detection_loss‚2.250703621123518 0loss/SRqqt4/detection_loss‚1.965404450893402 %loss/total_loss‚53.36267911166269 0loss/FRt1/assignment_loss‚2.5228931917084587 0loss/FBt3/assignment_loss‚0.2406421345862327 0loss/SRqqt3/detection_loss‚2.382913012057543 epoch‚1 2loss/SRqqt1/assignment_loss‚1.8840621748318276 1loss/FBt1/assignment_loss‚0.18873120859847645 /loss/FRt2/assignment_loss‚4.019762588398797 1loss/FBt4/assignment_loss‚0.26064372731462204 0loss/SRqqt2/detection_loss‚1.855889615687457 trainer/global_step‚1349 2loss/SRqqt4/assignment_loss‚2.0219502987650535 /loss/FBt2/detection_loss‚3.3017413285043506 /loss/FRt4/assignment_loss‚3.605644040781519 .loss/FRt3/detection_loss‚2.740482708439231qlÓ]Ù × 0loss/FRt1/assignment_loss‚2.5228931917084587 0loss/FBt3/assignment_loss‚0.2406421345862327 0loss/SRqqt3/detection_loss‚2.382913012057543 epoch‚1 2loss/SRqqt1/assignment_loss‚1.8840621748318276 1loss/FBt1/assignment_loss‚0.18873120859847645 /loss/FRt2/assignment_loss‚4.019762588398797 1loss/FBt4/assignment_loss‚0.26064372731462204 0loss/SRqqt2/detection_loss‚1.855889615687457 trainer/global_step‚1349 2loss/SRqqt4/assignment_loss‚2.0219502987650535 /loss/FBt2/detection_loss‚3.3017413285043506 /loss/FRt4/assignment_loss‚3.605644040781519 .loss/FRt3/detection_loss‚2.740482708439231 /loss/FBt3/detection_loss‚3.5255911469459535 0loss/SRqqt1/detection_loss‚1.352770004980266 /loss/FBt1/detection_loss‚0.8338085578755078 /loss/FBt4/detection_loss‚3.8034308661114085 _runtime‚1144.244494034 /loss/FRt3/assignment_loss‚5.722967103123665 2loss/SRqqt2/assignment_loss‚2.1103066384792326 /loss/FRt1/detection_loss‚1.9763471196095148 /loss/FRt4/detection_loss‚2.3349740842114324 % _timestamp‚1.7884387041995735e+09 _step‚110 2loss/SRqqt3/assignment_loss‚2.3326998241245747 1loss/FBt2/assignment_loss‚0.12831966451065252 .loss/FRt2/detection_loss‚2.250703621123518 0loss/SRqqt4/detection_loss‚1.965404450893402 %loss/total_loss‚53.36267911166269nÒ³bpŸÚ ‚(:” ºÑåÔÃá‹É gpu.0.gpu‚89.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚245.693' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.138600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n*D(Û ‚(: ºÑåÔô†´Ë disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1718.67578125 disk.dm-0.in‚ 1718.67578125 disk.drbd1777.out‚0) proc.memory.percent‚16.85171127319336 network.sent‚ 2.072272e+06 disk.rbd2.out‚ 65.423828125 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚13190.89697265625 cpu‚25.27660107930453 proc.cpu.threads‚17 disk.sda.out‚ 48.29296875 disk.sda2.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚13190.89697265625# disk.dm-0.out‚13190.89697265625$ disk./.usageGB‚1104.466136932373 disk.rbd1.out‚ 0.4873046875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚477688) disk./.usagePercent‚74.12164915834553! disk.nvme0n1.in‚ 1718.67578125 disk.sda3.in‚0 disk.sda3.out‚ 48.29296875 disk.rbd0.out‚0$ memory_percent‚77.91244983673096) proc.memory.availableMB‚ 3618.82421875! proc.memory.rssMB‚ 2760.984375 disk.dm-1.out‚ 48.29296875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0¡Û>ö”Ü ‚(:‰ ÉÑåÔÒª–É gpu.0.gpu‚70.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.488' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f M,Ý ‚(:¡ ÉÑåÔí¹õË disk.sda1.out‚0& disk.nvme0n1.out‚13290.64501953125 disk.sda3.in‚0 disk.dm-0.in‚ 1718.67578125$ memory_percent‚77.91125774383545! proc.memory.rssMB‚ 2760.984375 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 49.06640625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 3619.01953125 network.recv‚483847 disk.drbd1777.in‚0 disk.sda.out‚ 49.06640625 network.sent‚ 2.180954e+06% disk./.usageGB‚1104.4697036743164 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.5498046875! disk.nvme0n1.in‚ 1718.67578125 disk.rbd0.in‚0 disk.dm-1.out‚ 49.06640625 proc.cpu.threads‚17) disk./.usagePercent‚74.12188852539005 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1718.67578125( disk.nvme0c0n1.out‚13290.64501953125# disk.dm-0.out‚13290.64501953125) proc.memory.percent‚16.85171127319336 cpu‚25.273738604601515 disk.drbd1777.out‚0 disk.rbd2.out‚66.61572265625½o¡à2Þ - _wandbruntime‚1182  _runtime‚1182Ù×Åß ¿ _step‚111 % lr-AdamW/pg1‚0.000387222628037198 % lr-AdamW/pg2‚0.000387222628037198 trainer/global_step‚1399 % _timestamp‚1.7884387426608891e+09 _runtime‚1182.705523753;Ÿ€sÉà à % _timestamp‚1.7884387426608891e+09 _runtime‚1182.705523753 _step‚111 % lr-AdamW/pg1‚0.000387222628037198 % lr-AdamW/pg2‚0.000387222628037198 trainer/global_step‚1399ož¡)2á - _wandbruntime‚1182  _runtime‚1182Ä>%Mâ òO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26072341 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20624279 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158391955957760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20624273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136870738426368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/®1Ä«$inactive_split.all.peak‚1408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136870815055360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2592322086967 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23103923 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20624273 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46716652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13376371 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3567640918528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2594023942144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3567689413120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153321372261376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2594023942144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155913719776311 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727082 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚1182.706491076 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158394582557696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155797932015616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 347702272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26092373 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26092373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23103450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133303097507840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚155797958407168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158391955957760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153321397689344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3567640918528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2594023942144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3567689413120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153321372261376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2594023942144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155913719776311 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727082 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚1182.706491076 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158394582557696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155797932015616 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 347702272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26092373 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26092373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23103450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133303097507840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚155797958407168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158391955957760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153321397689344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2594151429843 $ _timestamp‚1.788438743397881e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26110633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2595856270848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3570204955648 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158504354605056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26090575 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23119805 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133395513695744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155905898112512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20638509 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153427738863616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13385567 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23119344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136965718651392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2598456492544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9733780 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155905898112512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136965642035712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153427713435648 trainer/global_step‚1399 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155905871720960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133395485561344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158501727991808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23119344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136965718651392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2598456492544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9733780 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155905898112512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136965642035712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153427713435648 trainer/global_step‚1399 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155905871720960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133395485561344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158501727991808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 WGDeviceStatsÂÒMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2594151429843 $ _timestamp‚1.788438743397881e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26110633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2595856270848 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3570204955648 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158504354605056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26090575 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23119805 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133395513695744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155905898112512 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20638509 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153427738863616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13385567 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376q¦DÏ®2ç - _wandbruntime‚1183  _runtime‚1183ç…Pè Ê 1loss/SRqqt2/assignment_loss‚2.064507018774748 0loss/FBt3/assignment_loss‚0.2297471926010783 $ _timestamp‚1.788438743399447e+09 /loss/FRt3/assignment_loss‚5.998308994151928 0loss/FBt1/assignment_loss‚0.1950781517023063 1loss/FBt2/assignment_loss‚0.49510901022170273 ,loss/FRt3/detection_loss‚2.6568939420912 1loss/SRqqt4/detection_loss‚1.8937016725540161 /loss/FRt1/detection_loss‚2.0164794756666473 epoch‚1 trainer/global_step‚1399 _step‚114 /loss/FRt4/assignment_loss‚4.080175833882026 1loss/FBt4/assignment_loss‚0.12613530503585935 1loss/SRqqt2/detection_loss‚1.9620234829684098 0loss/SRqqt3/detection_loss‚2.332401783125741 /loss/FBt2/detection_loss‚3.3238211699894498 /loss/FBt3/detection_loss‚3.1547130346298218 /loss/FRt2/assignment_loss‚4.241084739896986 ,loss/FBt4/detection_loss‚4.1337651014328 %loss/total_loss‚55.97487354009738 1loss/SRqqt3/assignment_loss‚2.904002593244825 .loss/FRt2/detection_loss‚2.537572325600518 .loss/FRt4/detection_loss‚2.457609104098014 1loss/SRqqt1/detection_loss‚1.4694666759069286 /loss/FBt1/detection_loss‚0.8328552582329172 _runtime‚1183.443998884 0loss/FRt1/assignment_loss‚2.8923996661020364 1loss/SRqqt1/assignment_loss‚1.758687968324147 1loss/SRqqt4/assignment_loss‚2.218334039863275Ùu Té Î /loss/FRt4/assignment_loss‚4.080175833882026 1loss/FBt4/assignment_loss‚0.12613530503585935 1loss/SRqqt2/detection_loss‚1.9620234829684098 0loss/SRqqt3/detection_loss‚2.332401783125741 /loss/FBt2/detection_loss‚3.3238211699894498 /loss/FBt3/detection_loss‚3.1547130346298218 /loss/FRt2/assignment_loss‚4.241084739896986 ,loss/FBt4/detection_loss‚4.1337651014328 %loss/total_loss‚55.97487354009738 1loss/SRqqt3/assignment_loss‚2.904002593244825 .loss/FRt2/detection_loss‚2.537572325600518 .loss/FRt4/detection_loss‚2.457609104098014 1loss/SRqqt1/detection_loss‚1.4694666759069286 /loss/FBt1/detection_loss‚0.8328552582329172 _runtime‚1183.443998884 0loss/FRt1/assignment_loss‚2.8923996661020364 1loss/SRqqt1/assignment_loss‚1.758687968324147 1loss/SRqqt4/assignment_loss‚2.218334039863275 1loss/SRqqt2/assignment_loss‚2.064507018774748 0loss/FBt3/assignment_loss‚0.2297471926010783 $ _timestamp‚1.788438743399447e+09 /loss/FRt3/assignment_loss‚5.998308994151928 0loss/FBt1/assignment_loss‚0.1950781517023063 1loss/FBt2/assignment_loss‚0.49510901022170273 ,loss/FRt3/detection_loss‚2.6568939420912 1loss/SRqqt4/detection_loss‚1.8937016725540161 /loss/FRt1/detection_loss‚2.0164794756666473 epoch‚1 trainer/global_step‚1399 _step‚114rÏÇí’ê ‚(:‡ ØÑåÔÏ•«É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚45.0 gpu.0.powerWatts‚274.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éä|+ë ‚(:  ØÑåÔºŸùË) proc.memory.percent‚16.85173511505127 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 1718.6796875 disk.sda.in‚0 disk.sda.out‚ 49.56640625 disk.rbd3.out‚0 disk.sda3.out‚ 49.56640625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1718.6796875 disk.rbd2.out‚67.62744140625# proc.memory.rssMB‚ 2760.98828125 cpu‚25.270762187098736 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.186709e+06) disk./.usagePercent‚74.12177639409006# disk.dm-0.out‚13292.51708984375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13292.51708984375 network.recv‚488215& disk.nvme0n1.out‚13292.51708984375 disk.dm-1.out‚ 49.56640625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚77.91216373443604$ disk./.usageGB‚1104.468032836914 disk.nvme0n1.in‚ 1718.6796875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.55029296875) proc.memory.availableMB‚ 3618.87109375ŒlöŸì ‚(:” çÑåÔ£ä¢É gpu.0.gpu‚70.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚250.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.062200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§)¯w*í ‚(:Ÿ çÑåÔÕ˜²Ë( disk.nvme0c0n1.out‚13391.73291015625 disk.rbd2.out‚67.73681640625# proc.memory.rssMB‚ 2760.98828125 cpu‚25.26829074637733 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 50.19921875 disk.sda2.out‚0& disk.nvme0n1.out‚13391.73291015625 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 50.19921875 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 2.295789e+06) disk./.usagePercent‚74.12177639409006 disk.dm-0.in‚ 1718.6796875# disk.dm-0.out‚13391.73291015625 disk.sda3.out‚ 50.19921875 network.recv‚494929$ disk./.usageGB‚1104.468032836914" disk.nvme0c0n1.in‚ 1718.6796875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 3618.49609375 disk.rbd1.out‚ 0.55029296875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1718.6796875$ memory_percent‚77.91445255279541) proc.memory.percent‚16.85173511505127 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0Ñ8PÜžî ‚(:“ öÑåÔ¯„‰É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚231.36' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.272000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ĺ ï ‚(:• öÑåÔÉœ•Ë disk.sda3.in‚0' disk.nvme0c0n1.out‚13491.2294921875 disk.dm-0.in‚ 1718.6796875 disk.rbd1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2760.98828125 disk.nvme0n1.in‚ 1718.6796875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚68.81103515625) proc.memory.percent‚16.85173511505127 network.sent‚ 2.301092e+06" disk.dm-0.out‚13491.2294921875 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 50.8984375 disk.sda1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1718.6796875 disk.rbd1.out‚ 0.61328125 cpu‚25.26546231250984 network.recv‚498785$ disk./.usageGB‚1104.468032836914 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 3618.5859375 proc.cpu.threads‚17% disk.nvme0n1.out‚13491.2294921875 disk.sda3.out‚ 50.8984375 disk.sda2.in‚0$ memory_percent‚77.91390419006348) disk./.usagePercent‚74.12177639409006 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 50.8984375 disk.drbd1777.in‚0h¡>2ð - _wandbruntime‚1222  _runtime‚1222zjü°Çñ Á trainer/global_step‚1449 % _timestamp‚1.7884387821756973e+09 _runtime‚1222.220383436 _step‚115 & lr-AdamW/pg1‚0.0003870491097013645 & lr-AdamW/pg2‚0.0003870491097013645»¨¹Ëò Å trainer/global_step‚1449 % _timestamp‚1.7884387821756973e+09 _runtime‚1222.220383436 _step‚115 & lr-AdamW/pg1‚0.0003870491097013645 & lr-AdamW/pg2‚0.0003870491097013645sÖt2ó - _wandbruntime‚1222  _runtime‚1222cÜßXù'ô óO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161319635501891 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21336079 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158638456402944 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21336073 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23899129 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163880572073984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10063433 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26987393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2685654791168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141616130169344 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161194943674368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137922528317440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2681204526915 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161194943674368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3693601851904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161322259214519 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074ÔK_‡õ ÷O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141616051443200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161322259214519 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161319635501891 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21336079 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158638456402944 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21336073 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23899129 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163880572073984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10063433 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26987393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2685654791168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141616130169344 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161194943674368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137922528317440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2681204526915 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161194943674368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3693601851904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 $ _timestamp‚1.788438782911242e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚163992970721280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23915017 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21350309 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163990344108032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161430429731667 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3696107825664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1222.956347432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10070121 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10069653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161302883379712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21350309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3696057247232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27005627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141711024196608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138014916370944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138014888236544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2683033869583 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158744797577216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163992970721280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13845364 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141710945483776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21350315 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161302856988160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158744772149248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23915488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13845367 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2690087341568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225Oçc 'ø N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2683033869583 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158744797577216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163992970721280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13845364 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141710945483776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21350315 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161302856988160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158744772149248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23915488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13845367 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2690087341568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2685632154451 _step‚117 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161302883379712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27025685 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27025685 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 $ _timestamp‚1.788438782911242e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚163992970721280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23915017 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21350309 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163990344108032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161430429731667 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3696107825664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1222.956347432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10070121 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10069653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161302883379712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21350309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3696057247232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27005627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141711024196608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48355936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138014916370944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138014888236544ueã"2ù - _wandbruntime‚1222  _runtime‚1222ìqiRú Ì 0loss/FBt2/assignment_loss‚0.5073485268013818 0loss/FBt4/assignment_loss‚0.7323699817059947 .loss/FRt2/detection_loss‚2.252932562320321 /loss/FRt3/detection_loss‚2.6353353303054283 epoch‚1 1loss/SRqqt1/assignment_loss‚1.852693886276992 2loss/SRqqt2/assignment_loss‚1.9162729726387904 /loss/FRt1/detection_loss‚1.9070245500492014 .loss/FBt4/detection_loss‚3.035725563764572 /loss/FRt1/assignment_loss‚2.678666801556297 /loss/FRt3/assignment_loss‚5.776880831554018 0loss/FBt3/assignment_loss‚0.2576815966167487 0loss/SRqqt3/detection_loss‚2.692874028827205 _step‚118 1loss/SRqqt1/detection_loss‚1.2173195062753033 /loss/FBt1/detection_loss‚0.9329137198774983 trainer/global_step‚1449 _runtime‚1222.957591366 1loss/SRqqt4/assignment_loss‚2.110519020093812 1loss/SRqqt4/detection_loss‚2.2642560722651304 &loss/total_loss‚55.100641231322825 /loss/FRt4/assignment_loss‚4.634127644130162 0loss/FBt1/assignment_loss‚0.2562588285571622 .loss/FRt4/detection_loss‚2.749256554671696 -loss/FBt3/detection_loss‚4.62382809817791 $ _timestamp‚1.788438782912988e+09 0loss/FRt2/assignment_loss‚3.6503191546157554 2loss/SRqqt3/assignment_loss‚2.0364188371282634 0loss/SRqqt2/detection_loss‚2.003521680831909 .loss/FBt2/detection_loss‚2.376095482281276ùÅÙVû Ð /loss/FRt3/assignment_loss‚5.776880831554018 0loss/FBt3/assignment_loss‚0.2576815966167487 0loss/SRqqt3/detection_loss‚2.692874028827205 _step‚118 1loss/SRqqt1/detection_loss‚1.2173195062753033 /loss/FBt1/detection_loss‚0.9329137198774983 trainer/global_step‚1449 _runtime‚1222.957591366 1loss/SRqqt4/assignment_loss‚2.110519020093812 1loss/SRqqt4/detection_loss‚2.2642560722651304 &loss/total_loss‚55.100641231322825 /loss/FRt4/assignment_loss‚4.634127644130162 0loss/FBt1/assignment_loss‚0.2562588285571622 .loss/FRt4/detection_loss‚2.749256554671696 -loss/FBt3/detection_loss‚4.62382809817791 $ _timestamp‚1.788438782912988e+09 0loss/FRt2/assignment_loss‚3.6503191546157554 2loss/SRqqt3/assignment_loss‚2.0364188371282634 0loss/SRqqt2/detection_loss‚2.003521680831909 .loss/FBt2/detection_loss‚2.376095482281276 0loss/FBt2/assignment_loss‚0.5073485268013818 0loss/FBt4/assignment_loss‚0.7323699817059947 .loss/FRt2/detection_loss‚2.252932562320321 /loss/FRt3/detection_loss‚2.6353353303054283 epoch‚1 1loss/SRqqt1/assignment_loss‚1.852693886276992 2loss/SRqqt2/assignment_loss‚1.9162729726387904 /loss/FRt1/detection_loss‚1.9070245500492014 .loss/FBt4/detection_loss‚3.035725563764572 /loss/FRt1/assignment_loss‚2.678666801556297v±æŸü ‚(:” …ÒåÔñ¡É gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.444199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+kõ–"ý ‚(:— …ÒåÔÒË•Ë) proc.memory.availableMB‚ 3619.02734375 disk.sda3.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚13590.3544921875 disk.dm-1.out‚ 51.7265625 disk.sda.out‚ 51.7265625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 1718.6796875$ disk./.usageGB‚1104.468032836914 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚13590.3544921875 disk.dm-1.in‚0) disk./.usagePercent‚74.12177639409006 disk.sda.in‚0 disk.nvme0n1.in‚ 1718.6796875 disk.rbd0.out‚0 cpu‚25.262890757408474 network.recv‚502935 disk.sda2.out‚0 disk.sda3.out‚ 51.7265625 disk.rbd1.in‚0' disk.nvme0c0n1.out‚13590.3544921875 network.sent‚ 2.307042e+06 disk.rbd2.out‚70.06494140625 disk.rbd0.in‚0 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2760.98828125) proc.memory.percent‚16.85173511505127 disk.rbd2.in‚0 disk.dm-0.in‚ 1718.6796875 disk.drbd1777.out‚0$ memory_percent‚77.91121006011963n¾‚”þ ‚(:‰ ”ÒåÔÆôôÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚46.0 gpu.0.powerWatts‚149.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ#ÿ ‚(: ”ÒåÔ˜‰¦Ë network.sent‚ 2.415424e+06 disk.sda2.in‚0 disk.sda3.out‚ 52.2109375 disk.rbd0.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2760.98828125% disk./.usageGB‚1104.4680366516113 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1718.6796875% disk.nvme0c0n1.out‚13689.54296875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 1718.6796875 proc.cpu.threads‚17 disk.sda1.in‚0( proc.memory.availableMB‚ 3618.9765625) proc.memory.percent‚16.85173511505127 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0) disk./.usagePercent‚74.12177665009759 disk.sda.in‚0 disk.sda.out‚ 52.2109375 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚13689.54296875 cpu‚25.260374903724774 disk.dm-1.out‚ 52.2109375 disk.rbd1.out‚ 0.61328125 network.recv‚509421# disk.nvme0n1.out‚13689.54296875 disk.dm-0.in‚ 1718.6796875 disk.rbd2.out‚ 71.1396484375$ memory_percent‚77.91152000427246»8Ã’€‚(:‡ £ÒåÔ²üºÉ gpu.0.gpu‚69.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚233.79' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.758 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÆÔ‚(: £ÒåÔäÆíË) disk./.usagePercent‚74.12177613808252% disk./.usageGB‚1104.4680290222168 disk.sda3.in‚0 disk.dm-0.out‚ 13689.9609375 disk.sda.in‚0 disk.sda.out‚ 52.83203125 disk.rbd3.out‚0 disk.dm-0.in‚ 1718.6796875 disk.rbd2.in‚0" proc.memory.rssMB‚ 2760.9921875 disk.sda2.out‚0 disk.rbd3.in‚0) proc.memory.percent‚16.85175895690918 network.recv‚513889 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 13689.9609375 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.61328125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 1718.6796875$ memory_percent‚99.95133876800537 cpu‚25.257900025373427 disk.nvme0n1.in‚ 1718.6796875 disk.sda3.out‚ 52.83203125 disk.drbd1777.out‚0 network.sent‚ 2.421742e+06 disk.rbd0.out‚0 disk.dm-1.out‚ 52.83203125 disk.rbd1.in‚0& proc.memory.availableMB‚ 7.97265625 disk.sda1.in‚0" disk.nvme0n1.out‚ 13689.9609375 disk.dm-1.in‚0 disk.rbd2.out‚ 71.1708984375Uf€2‚- _wandbruntime‚1261  _runtime‚1261ë˜âɃà trainer/global_step‚1499 % _timestamp‚1.7884388216561406e+09 _runtime‚1261.700813026 _step‚119 ' lr-AdamW/pg1‚0.00038683530958938254 ' lr-AdamW/pg2‚0.00038683530958938254, Í„Ç trainer/global_step‚1499 % _timestamp‚1.7884388216561406e+09 _runtime‚1261.700813026 _step‚119 ' lr-AdamW/pg1‚0.00038683530958938254 ' lr-AdamW/pg2‚0.00038683530958938254w®É š2…- _wandbruntime‚1261  _runtime‚1261N'äUâ †òO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 _step‚120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2777285640192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22047879 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10398844 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146361270311424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 347702272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163955515116544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2777285640192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27902445 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27922477 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27902445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14295971 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 $ _timestamp‚1.788438821656926e+09 E8DeviceStatsMonitor.on_train_batch_start/active.a]Žà±ll.freed‚49950318 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24694356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2772685251523 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166730798652727 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169369188190208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779885848576 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142541902858240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1261.70186046 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169369188190208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3819290824192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146361193682432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22047879 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142541930992640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166728174940099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22047873 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166591928941568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166591902550016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3819339318784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2779885848576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10398388 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169371814790144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24694815 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163955489688576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22047873 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166591928941568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166591902550016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3819339318784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23666360320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2779885848576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10398388 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169371814790144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24694815 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163955489688576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166730798652727 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169369188190208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779885848576 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142541902858240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 1261.70186046 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169369188190208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3819290824192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146361193682432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22047879 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0x\!àÒ2ˆ- _wandbruntime‚1262  _runtime‚1262i3»ö'‰þM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2781718190592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3821814793216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166836345457039 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166699842255360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169481586837504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146456182320640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚1499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166699868646912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22062115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166699868646912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27940737 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24710295 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781718190592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14305167 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169481586837504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2777112879059 % _timestamp‚1.7884388223958194e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2774514594191 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2779117968896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27920679 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24710727 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 _runtime‚1262.441142174 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22062109 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166699842255360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146456105704960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142634290911744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27940737 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164061830862848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚1499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166699868646912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23666360320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22062115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166699868646912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27940737 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24710295 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781718190592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14305167 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169481586837504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2777112879059 % _timestamp‚1.7884388223958194e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2774514594191 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 XFDeviceStatsMonitor.8íK,son_train_batch_end/allocated_bytes.small_pool.freed‚ 2779117968896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27920679 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24710727 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 _runtime‚1262.441142174 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22062109 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166699842255360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146456105704960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142634290911744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27940737 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164061830862848 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 E´2•- _wandbruntime‚1301  _runtime‚1301¹Uãù'–óO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2868916489216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2871516697600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172139338090935 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 $ _timestamp‚1.788438861133191e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174857804306432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172136714378307 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10732862 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169272548402176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2866764260791 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3945119071744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174860430906368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531840 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22759679 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3945165469184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2868916489216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10732409 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171988887817216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25488633 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22759679 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169272573830144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51597208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151106499137024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174857804306432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46397440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚1549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174860430906368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28837529 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2864165976131 _runtime‚1301.178115636 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28837529 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28817497 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚171988914208768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25488177 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809G~¼ý'—÷O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2868916489216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2871516697600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172139338090935 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 $ _timestamp‚1.788438861133191e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174857804306432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172136714378307 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10732862 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23666360320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169272548402176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2866764260791 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3945119071744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2646 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174860430906368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531840 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22759679 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3945165469184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2868916489216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10732409 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171988887817216 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25488633 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22759679 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169272573830144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51597208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151106499137024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174857804306432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46397440 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚1549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174860430906368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28837529 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2864165976131 _runtime‚1301.178115636 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28837529 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28817497 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚171988914208768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25488177 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646|æMrÊ2˜- _wandbruntime‚1301  _runtime‚1301ºªuF·™ýM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174970202953728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.currentqXL$‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2865995318799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22773915 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 _runtime‚1301.917520643 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22773909 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚1549 $ _timestamp‚1.788438861872216e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147253693586944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2868593603667 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51609640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14764964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2870748817920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2870748817920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28835731 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151201405122048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28855789 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2873349039616 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172247508608083 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174970202953728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2873349039616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46384128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3947637016576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25504067 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3947683400704 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28855789 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2873349039616 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172247508608083 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23666360320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174970202953728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2873349039616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46384128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3947637016576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25504067 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3947683400704 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2865995318799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22773915 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2646 _runtime‚1301.917520643 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22773909 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚1549 $ _timestamp‚1.788438861872216e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3296722944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147253693586944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2868593603667 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51609640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14764964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704}’:ë€2›- _wandbruntime‚1301  _runtime‚1301R¿÷/VœÐ /loss/FRt1/assignment_loss‚2.343845257759094 .loss/FRt2/assignment_loss‚4.20948818901129 0loss/FBt4/assignment_loss‚0.5240962410769219 /loss/FRt4/detection_loss‚2.5399826724420893 1loss/SRqqt1/detection_loss‚1.0139170936957191 /loss/FRt3/assignment_loss‚5.543376532467929 _runtime‚1301.918660731 /loss/FRt4/assignment_loss‚4.086703612045809 1loss/FBt3/assignment_loss‚0.31217061224022774 /loss/FRt1/detection_loss‚2.0828198742866517 .loss/FBt3/detection_loss‚3.552011718352636 epoch‚1 $ _timestamp‚1.788438861874059e+09 1loss/FBt1/assignment_loss‚0.25135822581979345 0loss/FBt2/assignment_loss‚0.5404098927974701 1loss/SRqqt2/detection_loss‚1.6414058169493309 trainer/global_step‚1549 _step‚126 1loss/SRqqt1/assignment_loss‚1.474872443966473 0loss/SRqqt2/assignment_loss‚2.34192511508098 /loss/FRt3/detection_loss‚2.3783754001964224 1loss/SRqqt4/detection_loss‚1.9782430872969006 /loss/FBt1/detection_loss‚0.8251630994608333 .loss/FBt2/detection_loss‚4.413687169551849 &loss/total_loss‚54.123149533343934 2loss/SRqqt3/assignment_loss‚2.4968315431049892 2loss/SRqqt4/assignment_loss‚2.1265980474974797 /loss/FRt2/detection_loss‚2.1771389141417385 1loss/SRqqt3/detection_loss‚2.5784575598580495 .loss/FBt4/detection_loss‚2.6902714142432585‰“4ZÔ /loss/FRt4/assignment_loss‚4.086703612045809 1loss/FBt3/assignment_loss‚0.31217061224022774 /loss/FRt1/detection_loss‚2.0828198742866517 .loss/FBt3/detection_loss‚3.552011718352636 epoch‚1 $ _timestamp‚1.788438861874059e+09 1loss/FBt1/assignment_loss‚0.25135822581979345 0loss/FBt2/assignment_loss‚0.5404098927974701 1loss/SRqqt2/detection_loss‚1.6414058169493309 trainer/global_step‚1549 _step‚126 1loss/SRqqt1/assignment_loss‚1.474872443966473 0loss/SRqqt2/assignment_loss‚2.34192511508098 /loss/FRt3/detection_loss‚2.3783754001964224 1loss/SRqqt4/detection_loss‚1.9782430872969006 /loss/FBt1/detection_loss‚0.8251630994608333 .loss/FBt2/detection_loss‚4.413687169551849 &loss/total_loss‚54.123149533343934 2loss/SRqqt3/assignment_loss‚2.4968315431049892 2loss/SRqqt4/assignment_loss‚2.1265980474974797 /loss/FRt2/detection_loss‚2.1771389141417385 1loss/SRqqt3/detection_loss‚2.5784575598580495 .loss/FBt4/detection_loss‚2.690271414243258 /loss/FRt1/assignment_loss‚2.343845257759094 .loss/FRt2/assignment_loss‚4.20948818901129 0loss/FBt4/assignment_loss‚0.5240962410769219 /loss/FRt4/detection_loss‚2.5399826724420893 1loss/SRqqt1/detection_loss‚1.0139170936957191 /loss/FRt3/assignment_loss‚5.543376532467929 _runtime‚1301.918660731~& bû”ž‚(:‰ ÐÒåÔóÒ†É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚44.0 gpu.0.powerWatts‚173.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ßË럂(: ÐÒåÔÕž±Ë disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚13988.2158203125 disk.dm-1.in‚0 disk.dm-1.out‚ 54.40234375 network.sent‚ 2.541359e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚73.58154296875 disk.sda.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1718.6875 disk.rbd1.out‚ 0.61328125) proc.memory.percent‚16.85175895690918 proc.cpu.threads‚17 cpu‚25.250894318465917 network.recv‚528772) disk./.usagePercent‚74.12176640979622 disk.nvme0n1.in‚ 1718.6875 disk.sda3.out‚ 54.40234375 disk.nvme0c0n1.in‚ 1718.6875' disk.nvme0c0n1.out‚13988.2158203125" proc.memory.rssMB‚ 2760.9921875 disk.rbd3.out‚0% disk.nvme0n1.out‚13988.2158203125 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 7.15234375 disk.sda.out‚ 54.40234375% disk./.usageGB‚1104.4678840637207 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0# memory_percent‚99.9563455581665œPŸ ‚(:” ßÒåÔ• ùÈ gpu.0.gpu‚76.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.651' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.930200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›&ì¡‚(:Œ ßÒåÔÌØ°Ë disk.dm-1.out‚ 55.16796875 disk.rbd2.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚13988.5400390625" proc.memory.rssMB‚ 2760.9921875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚7.109375) proc.memory.percent‚16.85175895690918 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 55.16796875 disk.nvme0c0n1.in‚ 1718.6875 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚74.12176640979622% disk./.usageGB‚1104.4678840637207 disk.sda.out‚ 55.16796875 disk.sda3.in‚0 disk.sda2.out‚0 network.sent‚ 2.650438e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1718.6875 cpu‚25.248657298455267 disk.nvme0n1.in‚ 1718.6875% disk.nvme0n1.out‚13988.5400390625$ memory_percent‚99.95660781860352 network.recv‚535818 disk.rbd2.out‚74.62451171875" disk.dm-0.out‚13988.5400390625 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.61328125‰Ì’’¢‚(:‡ îÒåÔôÇ“É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.190994262695312* gpu.0.memoryAllocatedBytes‚ 24215879680 gpu.0.temp‚47.0 gpu.0.powerWatts‚227.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ¼g;£‚(:‘ îÒåÔ®îÊ network.sent‚ 2.655656e+06) disk./.usagePercent‚74.12176640979622 disk.drbd1777.in‚0& disk.nvme0n1.out‚14087.81884765625 disk.sda3.out‚ 55.72265625( disk.nvme0c0n1.out‚14087.81884765625 disk.dm-1.in‚0$ memory_percent‚99.95520114898682 network.recv‚539645% disk./.usageGB‚1104.4678840637207 disk.rbd1.out‚ 0.61328125 disk.sda1.out‚0 disk.sda3.in‚0# disk.dm-0.out‚14087.81884765625 disk.rbd2.out‚75.83935546875& proc.memory.availableMB‚ 7.33984375 disk.sda1.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2760.9921875 disk.nvme0n1.in‚ 1718.6875 disk.nvme0c0n1.in‚ 1718.6875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1718.6875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 55.72265625 disk.rbd0.in‚0) proc.memory.percent‚16.85175895690918 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 55.72265625 cpu‚25.246489441545446Þ>"-2¤- _wandbruntime‚1340  _runtime‚1340~¡Ç¥Á & lr-AdamW/pg1‚0.0003862870506448415 & lr-AdamW/pg2‚0.0003862870506448415 trainer/global_step‚1599 % _timestamp‚1.7884389004456587e+09 _runtime‚1340.490358376 _step‚127s£Vw˦Š% _timestamp‚1.7884389004456587e+09 _runtime‚1340.490358376 _step‚127 & lr-AdamW/pg1‚0.0003862870506448415 & lr-AdamW/pg2‚0.0003862870506448415 trainer/global_step‚1599ªI»g2§- _wandbruntime‚1340  _runtime‚1340‡1ò¡U¨óO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15215568 trainer/global_step‚1599 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180346420422656 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177385899475968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26284686 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 _runtime‚1340.491367608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23471473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2955646700739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29732549 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23471473 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2958244985399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2963147546624 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26285168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177385899475968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155851786465792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4071078257152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174589632543744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small—{ ¤_pool.peak‚1572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53204022 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2960547338240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177385873084416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180349047022592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180346420422656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23471479 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11069597 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23471479 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177385873084416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11069118 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 $ _timestamp‚1.788438900446489e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151780708208640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29752581ÇÍ8sþ'©øO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3296722944 _step‚128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53224060 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29732549 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29752581 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174589607115776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180349047022592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2963147546624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151780736343040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155851865191936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177547877529143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177545253816515 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23471473 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2958244985399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2963147546624 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26285168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177385899475968 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23666360320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155851786465792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4071078257152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174589632543744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53204022 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2960547338240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2646 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177385873084416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180349047022592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180346420422656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23471479 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11069597 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1572 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23471479 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177385873084416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11069118 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2646 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 $ _timestamp‚1.788438900446489e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151780708208640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3296722944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29752581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15215568 trainer/global_step‚1599 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180346420422656 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177385899475968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26284686 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3296722944 _runtime‚1340.491367608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2646 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23471473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2955646700739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29732549 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2668361728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1599 _runtime‚1341.236158657 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177493839181312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177493812789760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29750783 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29770841 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2964979888640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23485715 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151873124396544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 % _timestamp‚1.7884389011910505e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚129 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2962379666944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15224767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15224764 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155946770206208 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26300547 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2957476043407 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151873096262144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180458819069952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26301000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4073645809664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29770841 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29750783 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174695948290048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23485715 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1599 _runtime‚1341.236158657 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177493839181312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2668361728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177493812789760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29750783 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23666360320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29770841 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2964979888640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23485715 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151873124396544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 347702272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3296722944 % _timestamp‚1.7884389011910505e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚129 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2962379666944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15224767 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15224764 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155946770206208 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1572 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26300547 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2957476043407 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151873096262144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2646 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1572 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180458819069952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26301000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4073645809664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2646 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29770841 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3296722944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J=DeviceStatsMonitor.on_train_bat!2€©ch_end/active.small_pool.freed‚29750783 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174695948290048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23485715 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27079298 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15675371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156400139018240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24183279 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52688896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179906665829376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 trainer/global_step‚1649 _runtime‚1379.892517217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185835036538880 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179906691257344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3052178187264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27079783 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80823296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182953793254723 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30647601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30667633 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11403930 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185837663138816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182782858351616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11404412 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160596696987648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4196638792704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24183273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182782884743168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚485 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3052178187264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3054778395648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 % _timestamp‚1.7884389398476353e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3049725710007 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30647601 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182956416967351 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496Ât3ÿ'¹ùO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30647601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30667633 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11403930 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185837663138816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182782858351616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11404412 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160596696987648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4196638792704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24183273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182782884743168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚485 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3052178187264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3054778395648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 % _timestamp‚1.7884389398476353e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3049725710007 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30647601 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182956416967351 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27079298 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15675371 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156400139018240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24183279 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52688896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179906665829376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 trainer/global_step‚1649 _runtime‚1379.892517217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185835036538880 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179906691257344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3052178187264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27079783 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80823296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182953793254723 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896„»Æ𠜺‚(:‘ ›ÓåÔ·‡¥É gpu.0.gpu‚71.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.7' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.339999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹<}0»‚(:Ž ›ÓåÔ§î´Ë cpu‚25.240183919469477 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125$ disk./.usageGB‚1104.467887878418 disk.sda2.out‚0 disk.dm-1.out‚ 57.75390625 disk.drbd1777.in‚0& proc.memory.availableMB‚ 6.73046875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚14287.19091796875 disk.nvme0c0n1.in‚ 1718.6875) proc.memory.percent‚16.85175895690918 disk.drbd1777.out‚0 disk.sda.out‚ 57.75390625 disk.sda3.out‚ 57.75390625 disk.dm-0.in‚ 1718.6875# disk.dm-0.out‚14287.19091796875 disk.sda.in‚0 disk.nvme0n1.in‚ 1718.6875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2760.9921875) disk./.usagePercent‚74.12176666580375 disk.sda1.in‚0 disk.rbd2.out‚78.21142578125# memory_percent‚99.9589204788208 disk.rbd3.in‚0( disk.nvme0c0n1.out‚14287.19091796875 network.sent‚ 2.77551e+06 network.recv‚554136 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0|ÍÄ2¼- _wandbruntime‚1380  _runtime‚1380´s¯÷] ½ýM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80809984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4199145356288 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27095432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180013007003648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54883408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15684567 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24197509 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4199092680704 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3051555052883 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3048956768015 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PADeviceStatsMonitor.on_t ø—¦rain_batch_end/allocated_bytes.all.current‚ 2626613248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30665835 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156492527071744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182890798056960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24197515 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15684564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160691591618048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180013032431616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30685893 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185947435186176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182890798056960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156492498937344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3054010515968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3054010515968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 _step‚133 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3056610737664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11410865 % _timestamp‚1.7884389405846884e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30665835 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30685893 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27094958 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160691672428032 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183064587484499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 _runtime‚ 1380.62983325 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185947435186176 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3056610737664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182890824448512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27094958 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160691672428032 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183064587484499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 _runtime‚ 1380.62983325 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185947435186176 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3056610737664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182890824448512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27095432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180013007003648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54883408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15684567 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24197509 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4199092680704 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3051555052883 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3048956768015 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30665835 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156492527071744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182890798056960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24197515 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15684564 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160691591618048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180013032431616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30685893 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185947435186176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182890798056960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156492498937344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3054010515968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3054010515968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 _step‚133 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3056610737664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11410865 % _timestamp‚1.7884389405846884e+09…º]Å2¿- _wandbruntime‚1380  _runtime‚1380œŸ7YÀÓ /loss/FRt4/assignment_loss‚3.313027253965052 .loss/FBt1/detection_loss‚0.588858001784215 &loss/total_loss‚50.537310401161086 trainer/global_step‚1649 1loss/FBt2/assignment_loss‚0.49790474493056536 /loss/FRt2/detection_loss‚1.9432828090020589 /loss/FRt3/detection_loss‚2.4520162063486435 1loss/SRqqt3/detection_loss‚2.6646766374851096 _runtime‚1380.631413267 0loss/FRt2/assignment_loss‚3.9254438953740256 2loss/SRqqt1/assignment_loss‚1.8131887762305827 0loss/FBt4/assignment_loss‚0.5775665516654619 .loss/FRt4/detection_loss‚2.486607020947991 2loss/SRqqt2/assignment_loss‚2.1328335714583493 0loss/FBt3/assignment_loss‚0.8496385540347546 1loss/SRqqt2/detection_loss‚1.8538853063875316 /loss/FRt3/assignment_loss‚5.405486015712514 2loss/SRqqt4/assignment_loss‚1.8159927849968274 /loss/FBt3/detection_loss‚3.1312771290540695 % _timestamp‚1.7884389405868137e+09 2loss/SRqqt3/assignment_loss‚2.0473400508535318 0loss/FBt1/assignment_loss‚0.2717553957960533 .loss/FRt1/detection_loss‚1.831609522998333 1loss/SRqqt4/detection_loss‚1.9727387931197882 .loss/FBt2/detection_loss‚3.375791597366333 /loss/FBt4/detection_loss‚2.3313287049531937 0loss/FRt1/assignment_loss‚1.8856669783592224 /loss/SRqqt1/detection_loss‚1.36939409833688 epoch‚1 _step‚134ÝÖMÝ^ÁØ 2loss/SRqqt1/assignment_loss‚1.8131887762305827 0loss/FBt4/assignment_loss‚0.5775665516654619 .loss/FRt4/detection_loss‚2.486607020947991 2loss/SRqqt2/assignment_loss‚2.1328335714583493 0loss/FBt3/assignment_loss‚0.8496385540347546 1loss/SRqqt2/detection_loss‚1.8538853063875316 /loss/FRt3/assignment_loss‚5.405486015712514 2loss/SRqqt4/assignment_loss‚1.8159927849968274 /loss/FBt3/detection_loss‚3.1312771290540695 % _timestamp‚1.7884389405868137e+09 2loss/SRqqt3/assignment_loss‚2.0473400508535318 0loss/FBt1/assignment_loss‚0.2717553957960533 .loss/FRt1/detection_loss‚1.831609522998333 1loss/SRqqt4/detection_loss‚1.9727387931197882 .loss/FBt2/detection_loss‚3.375791597366333 /loss/FBt4/detection_loss‚2.3313287049531937 0loss/FRt1/assignment_loss‚1.8856669783592224 /loss/SRqqt1/detection_loss‚1.36939409833688 epoch‚1 _step‚134 /loss/FRt4/assignment_loss‚3.313027253965052 .loss/FBt1/detection_loss‚0.588858001784215 &loss/total_loss‚50.537310401161086 trainer/global_step‚1649 1loss/FBt2/assignment_loss‚0.49790474493056536 /loss/FRt2/detection_loss‚1.9432828090020589 /loss/FRt3/detection_loss‚2.4520162063486435 1loss/SRqqt3/detection_loss‚2.6646766374851096 _runtime‚1380.631413267 0loss/FRt2/assignment_loss‚3.9254438953740256†¬lÅe‚(:’ ªÓåÔÖ˜É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚97.207' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.441399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-UDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191323652655104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188179843618816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27874884 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3141206434615 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31562653 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188179843618816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24895079 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1419.407554072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24895073 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27874420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11739713 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 % _timestamp‚1.7884389793625252e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188179870010368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16135168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185223749970944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3146409244672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3143809036288 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16135171 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4322638358016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185223724542976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188364956405559 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191323652655104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188179870010368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3143809036288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165342103422464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161019541693440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3138608149955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 Z°"L5LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56457726 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31582685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161019513559040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1699 _step‚136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573©‘—ÿ'ËùO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31562653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4322589863424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31562653 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188179843618816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24895079 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1419.407554072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24895073 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27874420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11739713 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 % _timestamp‚1.7884389793625252e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188179870010368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16135168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185223749970944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3146409244672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3143809036288 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16135171 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4322638358016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185223724542976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188364956405559 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191323652655104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188179870010368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3143809036288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165342103422464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161019541693440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3138608149955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56457726 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31582685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161019513559040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1699 _step‚136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11739252 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191326279255040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191323652655104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188179843618816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27874884 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3141206434615 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188287809715712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3145641364992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11746331 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354018304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165437081318912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161111929746944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188473126922707 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16144367 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165437004703232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24909309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31600945 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27890253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31580887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16144364 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188287809715712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185330065717248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31580887 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191436051302400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24909309 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 _step‚137 % _timestamp‚1.7884389801043038e+09 _runtime‚1420.149618016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4325103090688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188470503209871 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56510260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191436051302400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191433424689152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0XO+Ó '΃N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31580887 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191436051302400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24909309 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 _step‚137 % _timestamp‚1.7884389801043038e+09 _runtime‚1420.149618016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4325103090688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188470503209871 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56510260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191436051302400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191433424689152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3148241586688 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188287809715712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3145641364992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11746331 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354018304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165437081318912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161111929746944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188473126922707 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16144367 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165437004703232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24909309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31600945 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27890253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31580887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16144364 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188287809715712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185330065717248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‰½kÏã2Ï- _wandbruntime‚1420  _runtime‚1420IYà­*ÐÓ /loss/FRt3/assignment_loss‚5.285956092178822 .loss/FBt3/detection_loss‚2.988601871899196 epoch‚1 0loss/FRt2/assignment_loss‚3.8279379956862507 0loss/FBt3/assignment_loss‚0.7929273971489498 1loss/SRqqt4/detection_loss‚1.8229445125217791 _step‚138 /loss/FRt4/assignment_loss‚4.653321628634994 1loss/SRqqt3/assignment_loss‚2.342531948378592 0loss/FBt4/assignment_loss‚0.8051938986671823 /loss/FRt1/detection_loss‚1.7879665957391262 /loss/FBt1/detection_loss‚0.6314344844454238 0loss/FRt1/assignment_loss‚1.8698244524002074 0loss/FBt1/assignment_loss‚0.2817231679295375 1loss/FBt2/assignment_loss‚0.46620899935563404 _runtime‚1420.150691299 % _timestamp‚1.7884389801061232e+09 1loss/SRqqt4/assignment_loss‚1.993453801782043 0loss/SRqqt1/detection_loss‚1.344197409730109 /loss/FBt2/detection_loss‚3.9618467489878335 /loss/FBt4/detection_loss‚2.6438203283718655 trainer/global_step‚1699 2loss/SRqqt1/assignment_loss‚1.4446530828163737 2loss/SRqqt2/assignment_loss‚2.×(‘Ø/1766167068204214 1loss/SRqqt2/detection_loss‚2.0711965533189995 %loss/total_loss‚53.47838632317046 .loss/FRt2/detection_loss‚2.315853041761062 /loss/FRt3/detection_loss‚2.5932755433022976 /loss/FRt4/detection_loss‚2.8423426392916085 1loss/SRqqt3/detection_loss‚2.5345574220021567`Ï×^ÑØ % _timestamp‚1.7884389801061232e+09 1loss/SRqqt4/assignment_loss‚1.993453801782043 0loss/SRqqt1/detection_loss‚1.344197409730109 /loss/FBt2/detection_loss‚3.9618467489878335 /loss/FBt4/detection_loss‚2.6438203283718655 trainer/global_step‚1699 2loss/SRqqt1/assignment_loss‚1.4446530828163737 2loss/SRqqt2/assignment_loss‚2.1766167068204214 1loss/SRqqt2/detection_loss‚2.0711965533189995 %loss/total_loss‚53.47838632317046 .loss/FRt2/detection_loss‚2.315853041761062 /loss/FRt3/detection_loss‚2.5932755433022976 /loss/FRt4/detection_loss‚2.8423426392916085 1loss/SRqqt3/detection_loss‚2.5345574220021567 /loss/FRt3/assignment_loss‚5.285956092178822 .loss/FBt3/detection_loss‚2.988601871899196 epoch‚1 0loss/FRt2/assignment_loss‚3.8279379956862507 0loss/FBt3/assignment_loss‚0.7929273971489498 1loss/SRqqt4/detection_loss‚1.8229445125217791 _step‚138 /loss/FRt4/assignment_loss‚4.653321628634994 1loss/SRqqt3/assignment_loss‚2.342531948378592 0loss/FBt4/assignment_loss‚0.8051938986671823 /loss/FRt1/detection_loss‚1.7879665957391262 /loss/FBt1/detection_loss‚0.6314344844454238 0loss/FRt1/assignment_loss‚1.8698244524002074 0loss/FBt1/assignment_loss‚0.2817231679295375 1loss/FBt2/assignment_loss‚0.46620899935563404 _runtime‚1420.150691299ŠD=Y!žÒ‚(:“ ÈÓåÔˆ¹ªÉ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚269.043' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.808600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¬YE!Ó‚(:– ÈÓåÔ½…÷Ë& proc.memory.availableMB‚ 6.69140625 cpu‚25.234442653468037( disk.nvme0c0n1.out‚14584.97314453125 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚570430 disk.sda.in‚0 disk.sda1.in‚0 disk.sda.out‚ 59.5546875 disk.dm-0.in‚ 1719.9453125 disk.rbd1.out‚ 0.61328125 disk.rbd3.out‚0% disk./.usageGB‚1104.4674644470215 disk.rbd2.in‚0 disk.sda3.out‚ 59.5546875 disk.dm-1.in‚0) proc.memory.percent‚16.85175895690918 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1719.9453125 disk.drbd1777.in‚0# memory_percent‚99.9591588973999" proc.memory.rssMB‚ 2760.9921875& disk.nvme0n1.out‚14584.97314453125# disk.dm-0.out‚14584.97314453125 disk.rbd3.in‚0 network.sent‚ 2.975656e+06) disk./.usagePercent‚74.12173824896745 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚80.67626953125 disk.nvme0n1.in‚ 1719.9453125 disk.dm-1.out‚ 59.5546875ˆÛÍ“Ô‚(:ˆ ×ÓåÔù½ãÈ gpu.0.gpu‚58.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚157.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ycé#Õ‚(:˜ ×ÓåÔ•ÌáË network.recv‚577435( disk.nvme0c0n1.out‚14684.10595703125# disk.dm-0.out‚14684.10595703125 disk.drbd1777.in‚0 disk.sda.out‚ 60.30859375 disk.sda2.in‚0 cpu‚25.232563264920497 disk.sda3.in‚0 disk.dm-0.in‚ 1719.9453125 disk.rbd2.out‚81.78173828125" proc.memory.rssMB‚ 2760.9921875 network.sent‚ 3.084565e+06% disk./.usageGB‚1104.4675254821777" disk.nvme0c0n1.in‚ 1719.9453125 disk.dm-1.out‚ 60.30859375 disk.sda.in‚0$ memory_percent‚99.95944499969482& proc.memory.availableMB‚ 6.64453125 disk.rbd1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.61328125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1719.9453125' disk./.usagePercent‚74.121742345088 disk.sda3.out‚ 60.30859375 disk.rbd3.out‚0& disk.nvme0n1.out‚14684.10595703125) proc.memory.percent‚16.85175895690918 disk.sda2.out‚0Ðõ±[Ö‚(:’ æÓåÔ¹ìÈ gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚228.414' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚45.68279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bÂv_ ׂ(:• æÓåÔñ€Ì$ proc.memory.availableMB‚7.328125) proc.memory.percent‚16.85175895690918 proc.cpu.threads‚17 network.sent‚ 3.089856e+06% disk./.usageGB‚1104.4675483703613 disk.rbd3.out‚0 disk.sda3.out‚ 61.04296875 disk.rbd0.out‚0 disk.rbd2.in‚0% disk.nvme0n1.out‚14685.4228515625$ memory_percent‚99.95527267456055" proc.memory.rssMB‚ 2760.9921875 cpu‚25.230537908672897 disk.rbd2.out‚82.93408203125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 1719.9453125" disk.dm-0.out‚14685.4228515625 disk.drbd1777.in‚0 disk.sda.out‚ 61.04296875 disk.rbd3.in‚0' disk.nvme0c0n1.out‚14685.4228515625 network.recv‚581241 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 61.04296875 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0) disk./.usagePercent‚74.12174388113321 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1719.9453125 disk.rbd0.in‚0 disk.dm-0.in‚ 1719.9453125$8¼Ô2Ø- _wandbruntime‚1458  _runtime‚1458ƒö”ŠÇÙÁ trainer/global_step‚1749 % _timestamp‚1.7884390184211683e+09 _runtime‚1458.465840319 _step‚139 & lr-AdamW/pg1‚0.0003851639350783201 & lr-AdamW/pg2‚0.00038516393507832010Œ¦ÌÚÆ trainer/global_step‚1749 % _timestamp‚1.7884390184211683e+09 _runtime‚1458.465840319 _step‚139 & lr-AdamW/pg1‚0.0003851639350783201 & lr-AdamW/pg2‚0.0003851639350783201‹PO%ž2Û- _wandbruntime‚1458  _runtime‚1458G4š™ú'ÜôO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190540783256576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3230088874563 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32497737 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32477705 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196812268771328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32497737 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25606873 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196814895371264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 % _timestamp‚1.7884390184220898e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28670938 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170087503113216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196814895371264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25606873 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193773495843767 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193770872131139 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25606879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193576828886016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3235439885312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170087426484224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165638916234240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4448558744576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 _runtime‚1458.467137243 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 trainer/global_step‚1749ã xAÿ'ÝùO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16594968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3238040093696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193576855277568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚193576828886016 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193576828886016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3235439885312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170087426484224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165638916234240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4448558744576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 _runtime‚1458.467137243 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 trainer/global_step‚1749 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190540783256576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3230088874563 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32497737 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32477705 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196812268771328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32497737 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25606873 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196814895371264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 % _timestamp‚1.7884390184220898e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28670938 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170087503113216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196814895371264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25606873 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193773495843767 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193770872131139 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25606879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32495939 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196924667418624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196924667418624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3237272214016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4451084866048 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3237272214016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3234516502099 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25621109 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3239872435712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170182417288192 ;5DeviceStatsMow Â×Snitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32515997 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12082542 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16604164 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165731332422144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58117048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3231918217231 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32515997 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚193684794982912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28686263 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193684768591360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196922040805376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193684768591360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46384128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32515997 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚193684794982912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28686263 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193684768591360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196922040805376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193684768591360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46384128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32495939 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196924667418624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196924667418624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3237272214016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4451084866048 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3237272214016 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3234516502099 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25621109 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3239872435712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170182417288192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32515997 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12082542 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647?Òš2á- _wandbruntime‚1459  _runtime‚1459Ú‡ZâÔ 1loss/FBt1/assignment_loss‚0.22484215613011188 0loss/FBt3/assignment_loss‚0.6239290608485278 0loss/FBt4/assignment_loss‚0.5949204000644386 /loss/FRt1/detection_loss‚2.0744910372988037 .loss/FRt2/detection_loss‚2.424479492008686 1loss/SRqqt3/detection_loss‚2.5249683260917664 2loss/SRqqt1/assignment_loss‚1.4889026095660833 2loss/SRqqt2/assignment_loss‚1.6023884498410754 %loss/total_loss‚49.53364048218148 0loss/FRt4/assignment_loss‚3.7625954437255857 1loss/SRqqt1/detection_loss‚1.2042872887391312 /loss/FRt3/assignment_loss‚4.814595019375837 2loss/SRqqt3/assignment_loss‚2.4349984149138133 .loss/FRt3/detection_loss‚2.177419499114708 1loss/SRqqt4/detection_loss‚2.0374461619750313 % _timestamp‚1.7884390191636453e+09 _runtime‚1459.208188674 /loss/FRt1/assignment_loss‚2.591416000024132 /loss/FRt4/detection_loss‚2.3476655459403992 /loss/FBt1/detection_loss‚0.6391790297528017 0loss/FRt2/assignment_loss‚3.7000969698031745 0loss/SRqqt2/detection_loss‚2.047835855682691 epoch‚1 2loss/SRqqt4/assignment_loss‚2.0436605330070723 /loss/FBt2/detection_loss‚2.5467090904712677 /loss/FBt4/detection_loss‚2.4048837795853615 trainer/global_step‚1749 _step‚142 0loss/FBt2/assignment_loss‚0.6930728486622684 .loss/FBt3/detection_loss‚2.528857469558716ËΚ _ãÙ /loss/FRt3/assignment_loss‚4.814595019375837 2loss/SRqqt3/assignment_loss‚2.4349984149138133 .loss/FRt3/detection_loss‚2.177419499114708 1loss/SRqqt4/detection_loss‚2.0374461619750313 % _timestamp‚1.7884390191636453e+09 _runtime‚1459.208188674 /loss/FRt1/assignment_loss‚2.591416000024132 /loss/FRt4/detection_loss‚2.3476655459403992 /loss/FBt1/detection_loss‚0.6391790297528017 0loss/FRt2/assignment_loss‚3.7000969698031745 0loss/SRqqt2/detection_loss‚2.047835855682691 epoch‚1 2loss/SRqqt4/assignment_loss‚2.0436605330070723 /loss/FBt2/detection_loss‚2.5467090904712677 /loss/FBt4/detection_loss‚2.4048837795853615 trainer/global_step‚1749 _step‚142 0loss/FBt2/assignment_loss‚0.6930728486622684 .loss/FBt3/detection_loss‚2.528857469558716 1loss/FBt1/assignment_loss‚0.22484215613011188 0loss/FBt3/assignment_loss‚0.6239290608485278 0loss/FBt4/assignment_loss‚0.5949204000644386 /loss/FRt1/detection_loss‚2.0744910372988037 .loss/FRt2/detection_loss‚2.424479492008686 1loss/SRqqt3/detection_loss‚2.5249683260917664 2loss/SRqqt1/assignment_loss‚1.4889026095660833 2loss/SRqqt2/assignment_loss‚1.6023884498410754 %loss/total_loss‚49.53364048218148 0loss/FRt4/assignment_loss‚3.7625954437255857 1loss/SRqqt1/detection_loss‚1.2042872887391312ŽBodH“ä‚(:ˆ õÓåÔÑߊÉ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.151' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.2302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'„$Uå‚(:“ õÓåÔúøÿÊ disk.rbd2.out‚82.96533203125 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1722.546875# proc.memory.availableMB‚6.78125 cpu‚25.22876376029601 disk.drbd1777.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 62.09765625 disk.rbd0.out‚0# disk.dm-0.out‚14787.22509765625) disk./.usagePercent‚74.12161843744143 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 62.09765625) proc.memory.percent‚16.85175895690918 network.recv‚585501 disk.rbd1.in‚0& disk.nvme0n1.out‚14787.22509765625( disk.nvme0c0n1.out‚14787.22509765625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 1722.546875 disk.dm-1.out‚ 62.09765625" proc.memory.rssMB‚ 2760.9921875% disk./.usageGB‚1104.4656791687012 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 network.sent‚ 3.095787e+06 disk.nvme0n1.in‚ 1722.546875$ memory_percent‚99.95861053466797reÝžæ‚(:“ „ÔåÔßµ²É gpu.0.gpu‚81.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.742' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.548399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t†-Ýç‚(: „ÔåÔïɨË disk.dm-1.out‚62.625 disk.drbd1777.out‚0$ memory_percent‚99.95896816253662 proc.cpu.threads‚17) disk./.usagePercent‚74.12164762230033 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2760.99609375) proc.memory.percent‚16.85178279876709 disk.sda2.in‚0( disk.nvme0c0n1.out‚14888.69775390625# disk.dm-0.out‚14888.69775390625 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚62.625 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 6.72265625 disk.rbd2.out‚ 84.2001953125 disk.sda.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1722.57421875& disk.nvme0n1.out‚14888.69775390625# disk.nvme0c0n1.in‚ 1722.57421875 network.sent‚ 3.20432e+06 disk.sda.out‚62.625 disk.dm-0.in‚ 1722.57421875% disk./.usageGB‚1104.4661140441895 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 cpu‚25.226979067512787 network.recv‚592042 disk.rbd1.in‚0“u“Ñ2è- _wandbruntime‚1497  _runtime‚1497Ô‡…ìÉéà % _timestamp‚1.7884390578511734e+09 _runtime‚1497.895859739 _step‚143 ' lr-AdamW/pg1‚0.00038470967304292484 ' lr-AdamW/pg2‚0.00038470967304292484 trainer/global_step‚1799&gòÎêÈ trainer/global_step‚1799 % _timestamp‚1.7884390578511734e+09 _runtime‚1497.895859739 _step‚143 ' lr-AdamW/pg1‚0.00038470967304292484 ' lr-AdamW/pg2‚0.00038470967304292484ç ›2ë- _wandbruntime‚1497  _runtime‚1497ªÁÐ×ú'ìôO VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33392757 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17054768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3321569599171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3329670942720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198973840544768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202303511487488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198973814153216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26318679 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202300884887552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174832583312896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170258347043840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33412789 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3324167883831 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1497.896981192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29465089 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12409871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚202303511487488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198973814153216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3327070734336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195857867398144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26318673 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33412789 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12410318 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199179411569347 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195857841970176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26318673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198973840544768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174832506683904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33392757 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4574187774464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26318679 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4574236269056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3329670942720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202300884887552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3327070734336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 % _timestamp‚1.7884390578520176e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560Lj’öíùO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3321569599171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3329670942720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198973840544768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202303511487488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198973814153216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26318679 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202300884887552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174832583312896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170258347043840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33412789 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3324167883831 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1497.896981192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29465089 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12409871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚202303511487488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198973814153216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3327070734336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195857867398144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26318673 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33412789 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12410318 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199179411569347 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195857841970176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26318673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198973840544768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174832506683904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33392757 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4574187774464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26318679 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4574236269056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59731468 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3329670942720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202300884887552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3327070734336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 % _timestamp‚1.7884390578520176e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33392757 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17054768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Ëö¨2î- _wandbruntime‚1498  _runtime‚1498ä=®!'ïüM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17063964 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚145 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202413283534848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199081780250112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202410656921600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174927409576448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195964208572416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195964183144448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33410991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199081753858560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3325997226707 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33431049 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3328903063040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199290205799123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29480467 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12416503 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3323398941839 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 1498.64747229 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3328903063040 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26332915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788439058602206e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3331503284736 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29480909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17063967 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170350706962944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199287582086287 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170350735097344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26332909 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199081753858560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174927486192128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4576702613504 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202413283534848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199081780250112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202410656921600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174927409576448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195964208572416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195964183144448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33410991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199081753858560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3325997226707 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33431049 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3328903063040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199290205799123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29480467 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12416503 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3323398941839 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 1498.64747229 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3328903063040 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26332915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788439058602206e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3331503284736 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29480909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17063967 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170350706962944‘S†³‹2ñ- _wandbruntime‚1498  _runtime‚1498VªùJGòÁ 0loss/FBt3/assignment_loss‚0.6580803015015342 .loss/FRt2/detection_loss‚2.284208651983513 % _timestamp‚1.7884390586043422e+09 /loss/FRt3/assignment_loss‚5.484255224466324 1loss/SRqqt2/assignment_loss‚1.980782242878428 2loss/SRqqt4/assignment_loss‚1.6022332004788848 0loss/FBt4/assignment_loss‚0.8606369485337795 /loss/FBt1/detection_loss‚0.7757843891991603 0loss/FRt2/assignment_loss‚3.6992772997550243 /loss/FRt3/detection_loss‚2.2774832795063653 0loss/SRqqt3/detection_loss‚2.837659946803389 /loss/FBt2/detection_loss‚1.8015737533569336 /loss/FRt1/detection_loss‚1.7941331593914236 .loss/FBt4/detection_loss‚2.721015079454942 trainer/global_step‚1799 2loss/SRqqt1/assignment_loss‚1.4369297896822293 0loss/FBt1/assignment_loss‚0.3084820933730538 .loss/FBt3/detection_loss‚2.529759873043407 0loss/FRt4/assignment_loss‚3.2356260973353717 1loss/SRqqt3/assignment_loss‚2.612723366967563 1loss/SRqqt4/detection_loss‚1.9093399637275272 _runtime‚1498.648909739 0loss/FRt1/assignment_loss‚2.1976969673278486 loss/FBt2/assignment_loss‚0 1loss/SRqqt1/detection_loss‚1.1152292714651786 1loss/SRqqt2/detection_loss‚1.8462673754624601 %loss/total_loss‚48.40107639691823 _step‚146 /loss/FRt4/detection_loss‚2.4318981212238935 epoch‚1VƒÝOLóÆ .loss/FBt4/detection_loss‚2.721015079454942 trainer/global_step‚1799 2loss/SRqqt1/assignment_loss‚1.4369297896822293 0loss/FBt1/assignment_loss‚0.3084820933730538 .loss/FBt3/detection_loss‚2.529759873043407 0loss/FRt4/assignment_loss‚3.2356260973353717 1loss/SRqqt3/assignment_loss‚2.612723366967563 1loss/SRqqt4/detection_loss‚1.9093399637275272 _runtime‚1498.648909739 0loss/FRt1/assignment_loss‚2.1976969673278486 loss/FBt2/assignment_loss‚0 1loss/SRqqt1/detection_loss‚1.1152292714651786 1loss/SRqqt2/detection_loss‚1.8462673754624601 %loss/total_loss‚48.40107639691823 _step‚146 /loss/FRt4/detection_loss‚2.4318981212238935 epoch‚1 0loss/FBt3/assignment_loss‚0.6580803015015342 .loss/FRt2/detection_loss‚2.284208651983513 % _timestamp‚1.7884390586043422e+09 /loss/FRt3/assignment_loss‚5.484255224466324 1loss/SRqqt2/assignment_loss‚1.980782242878428 2loss/SRqqt4/assignment_loss‚1.6022332004788848 0loss/FBt4/assignment_loss‚0.8606369485337795 /loss/FBt1/detection_loss‚0.7757843891991603 0loss/FRt2/assignment_loss‚3.6992772997550243 /loss/FRt3/detection_loss‚2.2774832795063653 0loss/SRqqt3/detection_loss‚2.837659946803389 /loss/FBt2/detection_loss‚1.8015737533569336 /loss/FRt1/detection_loss‚1.7941331593914236’E§ ”žô‚(:“ “ÔåÔþÀ›É gpu.0.gpu‚89.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.163' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.232600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-’I~õ‚(:™ “ÔåÔ³Ë±Ë disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0" disk.dm-0.out‚14988.2431640625 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2760.99609375 network.recv‚596526% disk.nvme0n1.out‚14988.2431640625& proc.memory.availableMB‚ 6.68359375# disk.nvme0c0n1.in‚ 1723.64453125 network.sent‚ 3.210521e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 85.3056640625! disk.nvme0n1.in‚ 1723.64453125 disk.dm-0.in‚ 1723.64453125 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.12170906410854) proc.memory.percent‚16.85178279876709 cpu‚25.22523807372089 proc.cpu.threads‚17' disk.nvme0c0n1.out‚14988.2431640625% disk./.usageGB‚1104.4670295715332 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 63.49609375 disk.dm-1.in‚0 disk.dm-1.out‚ 63.49609375 disk.sda.out‚ 63.49609375$ memory_percent‚99.9592C§Ù 0658111572-ð“ö‚(:ˆ ¢ÔåÔÙÊàÊ gpu.0.gpu‚71.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚260.021' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lsß$÷‚(:™ ¢ÔåÔÏ„žÌ disk.sda2.in‚0 disk.rbd3.in‚0 network.sent‚ 3.318394e+06) disk./.usagePercent‚74.12167424708389 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 64.05078125$ memory_percent‚99.95789527893066 network.recv‚602281 disk.drbd1777.out‚0 disk.rbd2.out‚86.53662109375 cpu‚25.223666438688664 proc.cpu.threads‚17$ disk./.usageGB‚1104.466510772705 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1723.64453125 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1723.64453125 disk.sda3.out‚ 64.05078125 disk.dm-1.out‚ 64.05078125 disk.rbd1.out‚ 0.61328125% proc.memory.availableMB‚ 6.8984375# proc.memory.rssMB‚ 2760.99609375' disk.nvme0c0n1.out‚14989.5712890625 disk.rbd2.in‚0) proc.memory.percent‚16.85178279876709% disk.nvme0n1.out‚14989.5712890625 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 1723.64453125" disk.dm-0.out‚14989.5712890625 disk.drbd1777.in‚0ùϲ’ø‚(:‡ ±ÔåÔé“þÈ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚279.785' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.957 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ;ìù‚(: ±ÔåÔ†»äÊ proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1724.5703125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2760.99609375 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda3.out‚64.5625# disk.dm-0.out‚15088.64892578125 disk.dm-1.in‚0 disk.rbd2.in‚0# memory_percent‚99.9598741531372) disk./.usagePercent‚74.12169370365649 disk.sda1.out‚0( disk.nvme0c0n1.out‚15088.64892578125 disk.dm-1.out‚64.5625) proc.memory.percent‚16.85178279876709 network.recv‚606606 disk.sda.out‚64.5625 disk.rbd1.out‚ 0.61328125 cpu‚25.22188188967788% disk./.usageGB‚1104.4668006896973 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd2.out‚86.56787109375 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚15088.64892578125" disk.nvme0c0n1.in‚ 1724.5703125 disk.dm-0.in‚ 1724.5703125 network.sent‚ 3.324312e+06 disk.rbd0.out‚0& proc.memory.availableMB‚ 6.57421875ÿÞ¡2ú- _wandbruntime‚1537  _runtime‚1537Þ‹ôÖÆûÀ ' lr-AdamW/pg1‚0.00038421561676647424 ' lr-AdamW/pg2‚0.00038421561676647424 trainer/global_step‚1849 # _timestamp‚1.78843909698315e+09 _runtime‚ 1537.02781024 _step‚147*³bËüÅ ' lr-AdamW/pg1‚0.00038421561676647424 ' lr-AdamW/pg2‚0.00038421561676647424 trainer/global_step‚1849 # _timestamp‚1.78843909698315e+09 _runtime‚ 1537.02781024 _step‚147“Þ´»2ý- _wandbruntime‚1537  _runtime‚1537Õ7ºú'þôO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 trainer/global_step‚1849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 _step‚148 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27030479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179577686125568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34327841 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204370799420416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30258419 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207792127603712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3421301791744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3421301791744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17514571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12743848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884390969839778e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204370825811968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174877721584640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4699964540928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201174900683776 _runtime‚1537.028963162 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179577764851712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204587951007555 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30257955 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17514568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3415648608439 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207789501003776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204590574720183 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207792127603712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207789501003776 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27030479 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34307809 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224+7—Ïÿ'ÿùO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34307809 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34327841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174877749719040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3418701583360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884390969839778e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204370825811968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174877721584640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4699964540928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201174900683776 _runtime‚1537.028963162 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179577764851712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204587951007555 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30257955 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17514568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3415648608439 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207789501003776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204590574720183 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207792127603712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207789501003776 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27030479 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34307809 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 trainer/global_step‚1849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 _step‚148 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27030479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179577686125568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34327841 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204370799420416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30258419 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207792127603712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3421301791744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3421301791744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17514571 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12743848 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27044715 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3417477951315 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3414879666447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354018304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4702531226624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27044709 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61370752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179672668999168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30273736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174970109638144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34346101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12749972 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17523767 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4702480648192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3420533912064 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204698745237331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207901899651072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204478739125760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204478765517312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34346101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174970137772544 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204478765517312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3420533912064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27044715 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204478739125760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27044709 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚1537.772267664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30274196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884390977269704e+09 _step‚149 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3423134133760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204696121524495 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204478739125760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204478765517312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34346101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174970137772544 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204478765517312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3420533912064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27044715 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204478739125760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27044709 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚1537.772267664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30274196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884390977269704e+09 _step‚149 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3423134133760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204696121524495 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27044715 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3417477951315 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3414879666447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354018304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4702531226624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27044709 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61370752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179672668999168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30273736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174970109638144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34346101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12749972 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17523767 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4702480648192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3420533912064 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573•ߨ Ï2ƒ- _wandbruntime‚1537  _runtime‚1537TÏÝV„Ð .loss/FBt1/detection_loss‚0.529740047727026 % _timestamp‚1.7884390977291017e+09 1loss/SRqqt1/assignment_loss‚1.639405742602449 2loss/SRqqt2/assignment_loss‚1.7753430402885049 /loss/FRt1/detection_loss‚2.1066126200285824 /loss/FBt3/detection_loss‚3.2452739562307085 epoch‚1 1loss/SRqqt1/detection_loss‚1.3389953343381344 /loss/FBt4/detection_loss‚3.3695006668567657 _step‚150 0loss/FBt1/assignment_loss‚0.2797611776874037 0loss/FBt4/assignment_loss‚0.5662590457668557 .loss/FBt2/detection_loss‚4.099536895751953 %loss/total_loss‚53.06608351494479 0loss/FRt2/assignment_loss‚3.7590842694044113 /loss/FRt3/assignment_loss‚5.121204888119417 0loss/FRt4/assignment_loss‚3.5781963698718013 .loss/FRt2/detection_loss‚2.405606255556146 0loss/SRqqt3/detection_loss‚2.484447031361716 _runtime‚1537.773781306 0loss/FBt2/assignment_loss‚0.2994707697071135 .loss/FRt4/detection_loss‚2.248000831020122 2loss/SRqqt3/assignment_loss‚2.2942833525793893 2loss/SRqqt4/assignment_loss‚1.9185416958433517 0loss/FBt3/assignment_loss‚0.4951089606745102 /loss/FRt3/detection_loss‚2.6645028836586895 0loss/SRqqt2/detection_loss‚1.904249132184659 trainer/global_step‚1849 0loss/FRt1/assignment_loss‚2.7813568806106392 0loss/SRqqt4/detection_loss‚2.161601667074447™xàÄ[…Õ 1loss/SRqqt1/assignment_loss‚1.639405742602449 2loss/SRqqt2/assignment_loss‚1.7753430402885049 /loss/FRt1/detection_loss‚2.1066126200285824 /loss/FBt3/detection_loss‚3.2452739562307085 epoch‚1 1loss/SRqqt1/detection_loss‚1.3389953343381344 /loss/FBt4/detection_loss‚3.3695006668567657 _step‚150 0loss/FBt1/assignment_loss‚0.2797611776874037 0loss/FBt4/assignment_loss‚0.5662590457668557 .loss/FBt2/detection_loss‚4.099536895751953 %loss/total_loss‚53.06608351494479 0loss/FRt2/assignment_loss‚3.7590842694044113 /loss/FRt3/assignment_loss‚5.121204888119417 0loss/FRt4/assignment_loss‚3.5781963698718013 .loss/FRt2/detection_loss‚2.405606255556146 0loss/SRqqt3/detection_loss‚2.484447031361716 _runtime‚1537.773781306 0loss/FBt2/assignment_loss‚0.2994707697071135 .loss/FRt4/detection_loss‚2.248000831020122 2loss/SRqqt3/assignment_loss‚2.2942833525793893 2loss/SRqqt4/assignment_loss‚1.9185416958433517 0loss/FBt3/assignment_loss‚0.4951089606745102 /loss/FRt3/detection_loss‚2.6645028836586895 0loss/SRqqt2/detection_loss‚1.904249132184659 trainer/global_step‚1849 0loss/FRt1/assignment_loss‚2.7813568806106392 0loss/SRqqt4/detection_loss‚2.161601667074447 .loss/FBt1/detection_loss‚0.529740047727026 % _timestamp‚1.7884390977291017e+09–·N›ãž†‚(:“ ÀÔåÔ£Œ…É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚250.601' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.120200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßЇ‚(:“ ÀÔåÔ¯£åË disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0 disk.nvme0n1.in‚ 1724.5703125% disk.nvme0n1.out‚15188.2255859375 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.5703125 disk.sda1.in‚0 disk.sda3.out‚ 65.0078125 disk.rbd0.in‚0) proc.memory.percent‚16.85178279876709% disk./.usageGB‚1104.4672889709473 disk.drbd1777.in‚0$ memory_percent‚99.95877742767334 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 cpu‚25.22020843689499 disk.dm-1.in‚0 disk.dm-1.out‚ 65.0078125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1724.5703125# proc.memory.rssMB‚ 2760.99609375 network.sent‚ 3.33011e+06 network.recv‚611134 disk.rbd2.out‚87.81005859375 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 6.75390625 disk.sda.in‚0 disk.sda.out‚ 65.0078125' disk.nvme0c0n1.out‚15188.2255859375 disk.drbd1777.out‚0) disk./.usagePercent‚74.12172647262088" disk.dm-0.out‚15188.2255859375 proc.cpu.threads‚17ÈT¨˜“ˆ‚(:ˆ ÏÔåÔŠ°ñÈ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚260.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Ãú𠉂(:• ÏÔåÔ܈ºË disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚15287.4404296875$ proc.memory.availableMB‚6.453125 cpu‚25.21845270698284 network.recv‚618230 disk.sda1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1724.5703125 disk.dm-0.in‚ 1724.5703125 disk.sda.out‚ 65.97265625 disk.sda3.in‚0$ memory_percent‚99.96061325073242 disk.rbd2.out‚88.91162109375 disk.sda1.out‚0% disk.nvme0n1.out‚15287.4404296875" disk.dm-0.out‚15287.4404296875 disk.dm-1.out‚ 65.97265625% disk./.usageGB‚1104.4674224853516 disk.sda.in‚0 disk.dm-1.in‚0 network.sent‚ 3.439346e+06 disk.rbd1.out‚ 0.61328125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 1724.5703125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 65.97265625# proc.memory.rssMB‚ 2760.99609375) proc.memory.percent‚16.85178279876709) disk./.usagePercent‚74.12173543288458å‘Q‘Š‚(:† ÞÔåÔíàÈ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚103.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:»í"‹‚(:— ÞÔåÔÇÏóË disk.rbd3.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2760.99609375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 66.6484375 disk.drbd1777.in‚0& proc.memory.availableMB‚ 6.49609375 network.recv‚622208 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15386.87744140625 disk.dm-0.in‚ 1724.5703125 disk.rbd1.out‚ 0.61328125 disk.rbd2.out‚89.00537109375 disk.sda3.out‚ 66.6484375" disk.nvme0c0n1.in‚ 1724.5703125 cpu‚25.217017134541493 disk.sda2.in‚0 disk.nvme0n1.in‚ 1724.5703125 disk.rbd3.out‚0 network.sent‚ 3.444536e+06( disk./.usagePercent‚74.1217333848243 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 66.6484375) proc.memory.percent‚16.85178279876709# disk.dm-0.out‚15386.87744140625 disk.dm-1.in‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚15386.87744140625 proc.cpu.threads‚17% disk./.usageGB‚1104.4673919677734 disk.drbd1777.out‚0 disk.sda1.in‚0$ memory_percent‚99.96035099029541 disk.sda1.out‚0UÎu2Œ- _wandbruntime‚1576  _runtime‚1576¾ˆU×Å¿ % lr-AdamW/pg2‚0.000383681869209556 trainer/global_step‚1899 % _timestamp‚1.7884391364064066e+09 _runtime‚1576.451106757 _step‚151 % lr-AdamW/pg1‚0.000383681869209556j7jfÊŽÄ trainer/global_step‚1899 % _timestamp‚1.7884391364064066e+09 _runtime‚1576.451106757 _step‚151 % lr-AdamW/pg1‚0.000383681869209556 % lr-AdamW/pg2‚0.000383681869209556—!¹ìZ2- _wandbruntime‚1576  _runtime‚1576*tJ©ú'ôO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206491959397376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213278117120000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4825341664256 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209767784687616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62985172 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27742273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206491984825344 trainer/global_step‚1899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213278117120000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179497124259840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 _runtime‚1576.452198995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184322415332352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27742279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35242893 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3510332432384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3507129333047 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35222861 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13079078 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184322494058496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213280743719936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13078607 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209767811079168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ªyÒé¤ ‘ùO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35222861 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3510332432384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35242893 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62965134 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35242893 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3510332432384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3507129333047 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35222861 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13079078 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184322494058496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213280743719936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13078607 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209767811079168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206491959397376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213278117120000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4825341664256 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209767784687616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62985172 PCDeviceStatsMonñk•ß[itor.on_train_batch_start/allocation.large_pool.freed‚27742273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206491984825344 trainer/global_step‚1899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213278117120000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179497124259840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 _runtime‚1576.452198995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184322415332352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27742279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354018304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17983567 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13085554 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827838361600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 trainer/global_step‚1899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210104660962703 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3512164761088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31069121 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31068669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209875724392960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35261153 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209875724392960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚206598300571648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27756515 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13085105 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184417378809344 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 $ _timestamp‚1.788439137150602e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35241095 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213390515767296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512164761088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27756509 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3506360391055 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210107284675539 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3514764982784 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3514764982784 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _runtime‚ 1577.1959253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35261153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209875750784512 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17983567 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13085554 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827838361600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 trainer/global_step‚1899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210104660962703 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3512164761088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31069121 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31068669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209875724392960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35261153 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209875724392960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚206598300571648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27756515 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13085105 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184417378809344 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 $ _timestamp‚1.788439137150602e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35241095 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213390515767296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512164761088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27756509 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3506360391055 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210107284675539 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3514764982784 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3514764982784 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _runtime‚ 1577.1959253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35261153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209875750784512 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31847790 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4950740865024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3598610057655 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28454073 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184116555069440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218769359836160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218766733236224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218769359836160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3596011772995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28454079 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3604563489792 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31848276 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚486 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13414105 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4950690273280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18434168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28454079 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚483 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13413622 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184116526935040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36157945 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3601963281408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36137913 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189067295934464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189067217208320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215405029883971 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215164796346368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3604563489792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884391757619114e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28454079 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚483 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13413622 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184116526935040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36157945 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3601963281408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36137913 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189067295934464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189067217208320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215405029883971 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215164796346368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3604563489792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884391757619114e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31847790 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4950740865024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3598610057655 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28454073 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184116555069440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218769359836160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218766733236224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218769359836160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3596011772995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647œAB÷U2¢- _wandbruntime‚1616  _runtime‚1616;’6M'£üM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚1616.554735163 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31863664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4953248307712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218876505270272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184208914988544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218879131883520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13420770 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28468315 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36156147 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18443364 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218879131883520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36176205 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13420300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36176205 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3597841115663 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215515824113747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 # _timestamp‚1.78843917650955e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189162191430656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3600439400531 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36156147 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3606395831808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184208943122944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3606395831808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28468315 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215272736051712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215272709660160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215513200400911Ç`:È|¤N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18443364 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚473 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218879131883520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36176205 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13420300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36176205 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3597841115663 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚215515824113747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 # _timestamp‚1.78843917650955e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189162191430656 F@DeviceSta\a…+‹tsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3600439400531 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36156147 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3606395831808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184208943122944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3606395831808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28468315 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215272736051712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215272709660160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215513200400911 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚1616.554735163 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31863664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4953248307712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218876505270272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184208914988544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218879131883520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13420770 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28468315 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36156147ºƒO2¥- _wandbruntime‚1616  _runtime‚1616âo„WX¦Ò 2loss/SRqqt1/assignment_loss‚1.3364733757208223 1loss/FBt2/assignment_loss‚0.39298831997439265 epoch‚1 trainer/global_step‚1949 _runtime‚1616.555887975 0loss/FBt4/assignment_loss‚0.5192236346603584 /loss/FRt3/detection_loss‚2.4116371174653373 .loss/FBt2/detection_loss‚3.179679661989212 0loss/FRt1/assignment_loss‚2.5739199874693885 1loss/SRqqt3/assignment_loss‚2.384068871086294 .loss/FRt4/detection_loss‚2.347566291372827 0loss/SRqqt3/detection_loss‚3.022315041585402 /loss/FRt2/assignment_loss‚4.253446342853399 1loss/SRqqt1/detection_loss‚1.0561171601647916 /loss/FBt1/detection_loss‚0.7554544995333562 /loss/FBt3/detection_loss‚2.6701284050941467 /loss/FRt3/assignment_loss‚6.019025365511577 0loss/FRt4/assignment_loss‚3.1449780819263866 .loss/FBt4/detection_loss‚2.148819863796234 _step‚158 2loss/SRqqt4/assignment_loss‚1.6816558042385925 1loss/SRqqt4/detection_loss‚1.8372238924105961 % _timestamp‚1.7884391765113053e+09 1loss/SRqqt2/detection_loss‚1.9347000279847313 2loss/SRqqt2/assignment_loss‚2.0835171993456636 /loss/FBt1/assignment_loss‚0.207566566895502 0loss/FBt3/assignment_loss‚0.3078253227286041 /loss/FRt1/detection_loss‚1.6136153810902645 /loss/FRt2/detection_loss‚2.3997184416422477 %loss/total_loss‚50.28166465654013+ÙÂ]§× 2loss/SRqqt2/assignment_loss‚2.0835171993456636 /loss/FBt1/assignment_loss‚0.207566566895502 0loss/FBt3/assignment_loss‚0.3078253227286041 /loss/FRt1/detection_loss‚1.6136153810902645 /loss/FRt2/detection_loss‚2.3997184416422477 %loss/total_loss‚50.28166465654013 2loss/SRqqt1/assignment_loss‚1.3364733757208223 1loss/FBt2/assignment_loss‚0.39298831997439265 epoch‚1 trainer/global_step‚1949 _runtime‚1616.555887975 0loss/FBt4/assignment_loss‚0.5192236346603584 /loss/FRt3/detection_loss‚2.4116371174653373 .loss/FBt2/detection_loss‚3.179679661989212 0loss/FRt1/assignment_loss‚2.5739199874693885 1loss/SRqqt3/assignment_loss‚2.384068871086294 .loss/FRt4/detection_loss‚2.347566291372827 0loss/SRqqt3/detection_loss‚3.022315041585402 /loss/FRt2/assignment_loss‚4.253446342853399 1loss/SRqqt1/detection_loss‚1.0561171601647916 /loss/FBt1/detection_loss‚0.7554544995333562 /loss/FBt3/detection_loss‚2.6701284050941467 /loss/FRt3/assignment_loss‚6.019025365511577 0loss/FRt4/assignment_loss‚3.1449780819263866 .loss/FBt4/detection_loss‚2.148819863796234 _step‚158 2loss/SRqqt4/assignment_loss‚1.6816558042385925 1loss/SRqqt4/detection_loss‚1.8372238924105961 % _timestamp‚1.7884391765113053e+09 1loss/SRqqt2/detection_loss‚1.9347000279847313žã ’¨‚(:‡ ‹ÕåÔÛúŒÉ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚95.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0")÷%©‚(:š ‹ÕåÔªžÍË% disk./.usageGB‚1104.4707832336426 disk.sda.in‚0& proc.memory.availableMB‚ 6.30859375) proc.memory.percent‚16.85178279876709 disk.rbd3.out‚0 disk.dm-0.in‚ 1724.5703125 disk.sda3.out‚ 68.37109375 disk.rbd1.out‚ 0.61328125# proc.memory.rssMB‚ 2760.99609375 cpu‚25.212159300784386 disk.dm-1.out‚ 68.37109375 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1724.5703125# memory_percent‚99.9614953994751 proc.cpu.threads‚17) disk./.usagePercent‚74.12196097552224 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15589.17138671875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚15589.17138671875 network.sent‚ 3.565165e+06# disk.dm-0.out‚15589.17138671875 disk.dm-1.in‚0 disk.sda.out‚ 68.37109375 disk.sda1.out‚0 disk.rbd3.in‚0 network.recv‚637522 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1724.5703125 disk.rbd2.out‚92.43115234375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0éçlœª‚(:‘ šÕåÔÿÛêÈ gpu.0.gpu‚60.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚272.77' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.55399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>›ïø!«‚(:– šÕåÔ»ÄüË disk.rbd1.in‚0% proc.memory.availableMB‚ 6.0078125 cpu‚25.21068412407857) disk./.usagePercent‚74.12197736000444 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1104.4710273742676& disk.nvme0n1.out‚15689.01318359375 disk.sda.out‚ 68.9296875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1724.5703125 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1724.5703125 disk.dm-1.in‚0 disk.rbd2.out‚92.57177734375 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚643182 disk.sda3.out‚ 68.9296875# disk.dm-0.out‚15689.01318359375 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚99.96333122253418) proc.memory.percent‚16.85178279876709 disk.sda3.in‚0( disk.nvme0c0n1.out‚15689.01318359375 network.sent‚ 3.673155e+06 disk.dm-0.in‚ 1724.5703125 disk.dm-1.out‚ 68.9296875 disk.rbd1.out‚ 0.61328125 disk.rbd2.in‚0# proc.memory.rssMB‚ 2760.99609375 proc.cpu.threads‚17RE𓬂(:ˆ ©ÕåÔ¥«™É gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚270.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æè¼­‚(:” ©ÕåÔ†¶ÎË disk.sda2.in‚0 disk.sda3.out‚ 69.796875 disk.dm-0.in‚ 1724.5703125# disk.dm-0.out‚15689.32568359375 disk.drbd1777.out‚0% proc.memory.availableMB‚ 6.0078125 proc.cpu.threads‚17) disk./.usagePercent‚74.12197736000444" disk.nvme0c0n1.in‚ 1724.5703125( disk.nvme0c0n1.out‚15689.32568359375 disk.dm-1.out‚ 69.796875$ memory_percent‚99.96333122253418 network.sent‚ 3.679021e+06 disk.sda.in‚0 disk.sda1.in‚0 cpu‚25.209406848585527% disk./.usageGB‚1104.4710273742676 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 0.61328125& disk.nvme0n1.out‚15689.32568359375 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2760.99609375 disk.rbd2.out‚93.61474609375) proc.memory.percent‚16.85178279876709 disk.dm-1.in‚0 network.recv‚647512 disk.rbd2.in‚0 disk.sda.out‚ 69.796875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 1724.5703125 disk.rbd0.out‚0DÖä2®- _wandbruntime‚1655  _runtime‚1655Iì bǯÁ trainer/global_step‚1999 % _timestamp‚1.7884392152090728e+09 _runtime‚1655.253763481 _step‚159 & lr-AdamW/pg1‚0.0003824957534315023 & lr-AdamW/pg2‚0.0003824957534315023mC*Ì°Æ trainer/global_step‚1999 % _timestamp‚1.7884392152090728e+09 _runtime‚1655.253763481 _step‚159 & lr-AdamW/pg1‚0.0003824957534315023 & lr-AdamW/pg2‚0.0003824957534315023Ÿ‡4äÇ2±- _wandbruntime‚1655  _runtime‚1655&†¡ú'²ôO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32641697 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29165873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3696194338816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚220561781613568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 _step‚160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32642171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29165879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188735929610240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚217126102252544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188735957744640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚220561781613568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚220561755222016 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚217126076824576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354333184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37052965 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224257975952384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29165873 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66238876 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224255349352448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220813569322179 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18893971 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3690090782263 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3687492497603 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3696194338816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224255349352448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5076356400128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13748200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884392152098064e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224257975952384 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3693594130432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13747729 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3693594130432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193812237515776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18893968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66218838 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37052965 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5076307905536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 trainer/global_step‚1999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37072997 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37072997 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220561755222016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224255349352448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5076356400128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13748200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884392152098064e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224257975952384 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3693594130432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13747729 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3693594130432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193812237515776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18893968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66218838 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37052965 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5076307905536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 trainer/global_step‚1999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37072997 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37072997 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220561755222016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚220561781613568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚220561755222016 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚217126076824576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354333184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37052965 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224257975952384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29165873 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66238876 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224255349352448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220813569322179 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18893971 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3690090782263 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 k±ï.2´- _wandbruntime‚1656  _runtime‚1656=\u'µþM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3698026680832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚220669721318912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193907210139648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224367747999744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37071199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66251308 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224365121386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29180115 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5078864341504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚217232443426816 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29180109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220921739839119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37091257 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29180109 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18903164 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚220669694927360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1656.000834771 _step‚161 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚220669694927360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3691920125139 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193907131426816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 trainer/global_step‚1999 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3689321840271 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3698026680832 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37071199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32657960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13754793 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220924363551955 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3695426459136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37091257 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188828345798144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224367747999744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32657515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188828317663744”ÊØ# '¶ƒN N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193907131426816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 trainer/global_step‚1999 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3689321840271 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3698026680832 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37071199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32657960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13754793 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220924363551955 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3695426459136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37091257 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188828345798144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224367747999744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32657515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188828317663744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3698026680832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚220669721318912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193907210139648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224367747999744 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37071199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66251308 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224365121386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29180115 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5078864341504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚217232443426816 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29180109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220921739839119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37091257 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29180109 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18903164 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚220669694927360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1656.000834771 _step‚161 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚220669694927360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3691920125139 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308¡Ævd2·- _wandbruntime‚1656  _runtime‚1656½NM»U¸Ï /loss/FRt2/assignment_loss‚4.073848799422935 /loss/FBt1/detection_loss‚0.6895928433216698 .loss/FBt3/detection_loss‚2.696305309023176 _runtime‚1656.001880852 /loss/FRt4/assignment_loss‚4.290356474538004 2loss/SRqqt2/assignment_loss‚1.8126844196927314 /loss/FBt3/assignment_loss‚0.486618201647486 1loss/SRqqt1/detection_loss‚0.9569067142663463 1loss/SRqqt4/detection_loss‚1.9966294004562053 1loss/SRqqt3/assignment_loss‚2.192264061082493 /loss/FRt3/detection_loss‚2.3512981451219983 epoch‚1 1loss/FBt1/assignment_loss‚0.20728857758173594 /loss/FRt2/detection_loss‚2.2495231882289604 1loss/SRqqt2/detection_loss‚1.8295740014782138 $ _timestamp‚1.788439215957314e+09 0loss/SRqqt4/assignment_loss‚1.83681768400872 1loss/SRqqt3/detection_loss‚2.0561077229001303 /loss/FRt3/assignment_loss‚5.199422438939412 /loss/FRt1/detection_loss‚1.8880535278055404 %loss/total_loss‚50.16140720759377 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚1.3712934397551562 -loss/FBt2/detection_loss‚3.48184601465861 /loss/FBt4/detection_loss‚2.4003326424530576 _step‚162 0loss/FRt1/assignment_loss‚2.1659331228997973 0loss/FBt2/assignment_loss‚0.5055244962180344 0loss/FBt4/assignment_loss‚0.5244202047054257 /loss/FRt4/detection_loss‚2.8987657773879265ç üZ¹Ô 1loss/FBt1/assignment_loss‚0.20728857758173594 /loss/FRt2/detection_loss‚2.2495231882289604 1loss/SRqqt2/detection_loss‚1.8295740014782138 $ _timestamp‚1.788439215957314e+09 0loss/SRqqt4/assignment_loss‚1.83681768400872 1loss/SRqqt3/detection_loss‚2.0561077229001303 /loss/FRt3/assignment_loss‚5.199422438939412 /loss/FRt1/detection_loss‚1.8880535278055404 %loss/total_loss‚50.16140720759377 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚1.3712934397551562 -loss/FBt2/detection_loss‚3.48184601465861 /loss/FBt4/detection_loss‚2.4003326424530576 _step‚162 0loss/FRt1/assignment_loss‚2.1659331228997973 0loss/FBt2/assignment_loss‚0.5055244962180344 0loss/FBt4/assignment_loss‚0.5244202047054257 /loss/FRt4/detection_loss‚2.8987657773879265 /loss/FRt2/assignment_loss‚4.073848799422935 /loss/FBt1/detection_loss‚0.6895928433216698 .loss/FBt3/detection_loss‚2.696305309023176 _runtime‚1656.001880852 /loss/FRt4/assignment_loss‚4.290356474538004 2loss/SRqqt2/assignment_loss‚1.8126844196927314 /loss/FBt3/assignment_loss‚0.486618201647486 1loss/SRqqt1/detection_loss‚0.9569067142663463 1loss/SRqqt4/detection_loss‚1.9966294004562053 1loss/SRqqt3/assignment_loss‚2.192264061082493 /loss/FRt3/detection_loss‚2.3512981451219983 epoch‚1¢yf–žº‚(:“ ¸ÕåÔ‘ȘÉ gpu.0.gpu‚59.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.921' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.584199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô»”"»‚(:— ¸ÕåÔ³¯¸Ë disk.rbd2.out‚94.82958984375) proc.memory.percent‚16.85178279876709 cpu‚25.20785102696685 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚ 3.684823e+06" disk.nvme0c0n1.in‚ 1724.5703125" disk.dm-0.out‚15788.9365234375& proc.memory.availableMB‚ 6.47265625 disk.rbd2.in‚0 disk.sda.out‚ 70.32421875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0$ memory_percent‚99.96049404144287# proc.memory.rssMB‚ 2760.99609375 network.recv‚651807 disk.nvme0n1.in‚ 1724.5703125 disk.sda3.out‚ 70.32421875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚15788.9365234375 disk.drbd1777.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚74.12197736000444 disk.rbd3.in‚0% disk.nvme0n1.out‚15788.9365234375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0% disk./.usageGB‚1104.4710273742676 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.5703125 disk.dm-1.out‚ 70.32421875 disk.drbd1777.in‚0 disk.sda2.out‚0x ‘¼‚(:† ÇÕåÔÛ¼šÉ gpu.0.gpu‚54.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚269.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×’J ½‚(:‘ ÇÕåÔß³Ë disk.dm-0.in‚ 1724.5703125# disk.dm-0.out‚15888.10107421875 disk.sda3.out‚ 71.046875 cpu‚25.20661160993897 proc.cpu.threads‚17 network.recv‚658785) disk./.usagePercent‚74.12197736000444% disk./.usageGB‚1104.4710273742676 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 1724.5703125 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 71.046875 disk.sda.out‚ 71.046875# proc.memory.availableMB‚6.21875# proc.memory.rssMB‚ 2760.99609375) proc.memory.percent‚16.85178279876709 disk.nvme0n1.in‚ 1724.5703125 disk.drbd1777.in‚0 disk.rbd2.in‚0 network.sent‚ 3.794046e+06 disk.rbd1.out‚ 0.61328125 disk.rbd2.out‚95.93505859375& disk.nvme0n1.out‚15888.10107421875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.96204376220703 disk.sda3.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15888.10107421875sÑ·õ2¾- _wandbruntime‚1694  _runtime‚1694sé§íÆ¿À & lr-AdamW/pg1‚0.0003818436323951173 & lr-AdamW/pg2‚0.0003818436323951173 trainer/global_step‚2049 $ _timestamp‚1.788439254492958e+09 _runtime‚1694.537661564 _step‚163ºŒiSËÀÅ _runtime‚1694.537661564 _step‚163 & lr-AdamW/pg1‚0.0003818436323951173 & lr-AdamW/pg2‚0.0003818436323951173 trainer/global_step‚2049 $ _timestamp‚1.788439254492958e+09£gdç32Á- _wandbruntime‚1694  _runtime‚1694¨}àI ÂôO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29877679 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225958740489216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 % _timestamp‚1.7884392544937472e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37968017 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354333184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222443160966144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29877673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229743965468672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29877679 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3787825187840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3787825187840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 H>DeviceStatsMonitor.on_train_batch_start/Y¬‚äallocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33437476 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 _step‚164 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198557378985984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37968017 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354333184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222443160966144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29877673 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229743965468672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29877679 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3787825187840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3787825187840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33437476 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 _step‚164 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198557378985984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5204495333376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3780802564879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚165 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67878160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3789657529856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193447748473344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222549476712448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3787057308160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14090789 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198652215672320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226066706586112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3787057308160 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29891915 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5204543814656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226330279277327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229856364115968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33453316 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193447720338944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226066706586112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198652292288000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226332902990163 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19362967 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884392552293377e+09=3§ŽÈƒN ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5204495333376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3780802564879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚165 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67878160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3789657529856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193447748473344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222549476712448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3787057308160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14090789 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198652215672320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226066706586112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3787057308160 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29891915 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5204543814656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226330279277327 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229856364115968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33453316 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193447720338944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226066706586112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198652292288000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226332902990163 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19362967 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884392552293377e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19362964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280¥©`€2É- _wandbruntime‚1695  _runtime‚1695øeUèUÊÏ /loss/FRt2/assignment_loss‚3.278994663901951 1loss/SRqqt1/assignment_loss‚1.390036049507447 .loss/FRt1/detection_loss‚1.878899405456402 1loss/SRqqt1/detection_loss‚1.1827328927926164 .loss/FBt2/detection_loss‚2.879840305873326 0loss/FBt1/assignment_loss‚0.2260998270214292 0loss/FBt2/assignment_loss‚0.1792545323925359 1loss/SRqqt2/detection_loss‚1.9178304739442527 1loss/SRqqt3/detection_loss‚2.7234589015996016 .loss/FBt3/detection_loss‚2.465436041355133 1loss/FBt3/assignment_loss‚0.47877846471965313 1loss/SRqqt4/detection_loss‚1.9512469493875317 /loss/FBt1/detection_loss‚1.0850523690677278 trainer/global_step‚2049 % _timestamp‚1.7884392552312372e+09 2loss/SRqqt2/assignment_loss‚2.3357090213719536 ,loss/FRt4/detection_loss‚2.5330650806427 $loss/total_loss‚49.6603426264256 0loss/FRt1/assignment_loss‚2.5735997706651688 /loss/FRt3/assignment_loss‚4.608875880241394 1loss/SRqqt3/assignment_loss‚1.980043726938742 1loss/FBt4/assignment_loss‚0.44446504264287795 /loss/FBt4/detection_loss‚2.7254014015197754 epoch‚1 /loss/FRt2/detection_loss‚2.2886954131333725 _step‚166 0loss/FRt4/assignment_loss‚3.3561323314905165 1loss/SRqqt4/assignment_loss‚2.053412649561377 /loss/FRt3/detection_loss‚3.1232814311981203 _runtime‚1695.275817905ðùY³ZËÔ 1loss/SRqqt4/detection_loss‚1.9512469493875317 /loss/FBt1/detection_loss‚1.0850523690677278 trainer/global_step‚2049 % _timestamp‚1.7884392552312372e+09 2loss/SRqqt2/assignment_loss‚2.3357090213719536 ,loss/FRt4/detection_loss‚2.5330650806427 $loss/total_loss‚49.6603426264256 0loss/FRt1/assignment_loss‚2.5735997706651688 /loss/FRt3/assignment_loss‚4.608875880241394 1loss/SRqqt3/assignment_loss‚1.980043726938742 1loss/FBt4/assignment_loss‚0.44446504264287795 /loss/FBt4/detection_loss‚2.7254014015197754 epoch‚1 /loss/FRt2/detection_loss‚2.2886954131333725 _step‚166 0loss/FRt4/assignment_loss‚3.3561323314905165 1loss/SRqqt4/assignment_loss‚2.053412649561377 /loss/FRt3/detection_loss‚3.1232814311981203 _runtime‚1695.275817905 /loss/FRt2/assignment_loss‚3.278994663901951 1loss/SRqqt1/assignment_loss‚1.390036049507447 .loss/FRt1/detection_loss‚1.878899405456402 1loss/SRqqt1/detection_loss‚1.1827328927926164 .loss/FBt2/detection_loss‚2.879840305873326 0loss/FBt1/assignment_loss‚0.2260998270214292 0loss/FBt2/assignment_loss‚0.1792545323925359 1loss/SRqqt2/detection_loss‚1.9178304739442527 1loss/SRqqt3/detection_loss‚2.7234589015996016 .loss/FBt3/detection_loss‚2.465436041355133 1loss/FBt3/assignment_loss‚0.47877846471965313¦;kݞ̂(:“ åÕåÔºÆ¢É gpu.0.gpu‚70.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚228.134' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.626799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0% Í‚(:‹ åÕåÔÅǼË disk.sda1.in‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 5.8984375" disk.nvme0c0n1.in‚ 1724.5703125% disk.nvme0c0n1.out‚15988.39453125 network.recv‚667806 disk.rbd1.out‚ 0.69921875 disk.sda.out‚ 71.984375 disk.sda2.in‚0) disk./.usagePercent‚74.12197736000444 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚99.96399879455566) proc.memory.percent‚16.85178279876709 disk.dm-0.out‚15988.39453125 disk.dm-1.in‚0% disk./.usageGB‚1104.4710273742676 disk.nvme0n1.in‚ 1724.5703125 disk.sda3.out‚ 71.984375 cpu‚25.203760466714332 disk.dm-0.in‚ 1724.5703125 network.sent‚ 3.895609e+06 disk.rbd3.in‚0# disk.nvme0n1.out‚15988.39453125 disk.sda3.in‚0 disk.dm-1.out‚ 71.984375# proc.memory.rssMB‚ 2760.99609375 disk.rbd2.in‚0 disk.rbd2.out‚97.20556640625 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0txö“΂(:ˆ ôÕåÔ¾ˆÃÉ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚112.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ª^rÏ‚(:‡ ôÕåÔ©š Ë disk.rbd3.in‚0 disk.nvme0n1.in‚ 1724.5703125$ disk.nvme0n1.out‚16087.509765625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1724.5703125& disk.nvme0c0n1.out‚16087.509765625 network.sent‚ 3.993319e+06% disk./.usageGB‚1104.4699745178223 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.76171875$ memory_percent‚99.96509552001953) proc.memory.percent‚16.85178279876709 network.recv‚673873# proc.memory.rssMB‚ 2760.99609375 disk.rbd2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda.out‚72.46875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.out‚72.46875 disk.dm-0.in‚ 1724.5703125 disk.rbd1.in‚0# proc.memory.availableMB‚5.71875 cpu‚25.20245068030245) disk./.usagePercent‚74.12190670192498 disk.dm-1.out‚72.46875 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚16087.509765625 disk.rbd2.out‚ 98.4287109375 disk.sda2.out‚0y3 2Ð- _wandbruntime‚1734  _runtime‚1734PfÈÑ ' lr-AdamW/pg1‚0.00038115231439625236 ' lr-AdamW/pg2‚0.00038115231439625236 trainer/global_step‚2099 % _timestamp‚1.7884392939773016e+09 _runtime‚ 1734.02194829 _step‚167Ö%ÆÍÒÇ ' lr-AdamW/pg1‚0.00038115231439625236 ' lr-AdamW/pg2‚0.00038115231439625236 trainer/global_step‚2099 % _timestamp‚1.7884392939773016e+09 _runtime‚ 1734.02194829 _step‚167§ ªC2Ó- _wandbruntime‚1734  _runtime‚1734˜Ÿú'ÔôO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235232581584896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19813571 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3876855828480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 trainer/global_step‚2099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3873052231479 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14420048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197974734960640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 % _timestamp‚1.7884392939780426e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3879456036864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231355725756416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227760194251776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38903101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38883069 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235235208184832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38883069 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203302391927808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69472542 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30589479 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14419557 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 _step‚168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231355752147968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235235208184832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231355725756416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231355752147968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354333184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30589473 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5327628832768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5327578241024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69492580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3879456036864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚494 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231630648198595 _runtime‚1734.022955428 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235232581584896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34233125 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197974763095040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 trainer/global_step‚2099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3873052231479 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14420048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197974734960640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 % _timestamp‚1.7884392939780426e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3879456036864 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231355725756416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227760194251776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38903101 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38883069 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235235208184832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38883069 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203302391927808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69472542 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30589479 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14419557 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 _step‚168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231355752147968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235235208184832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231355725756416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚231355752147968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 354333184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30589473 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5327628832768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5327578241024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69492580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3879456036864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚494 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231630648198595 _runtime‚1734.022955428 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235232581584896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34233125 TEDeviceStatsMonitor.on_train_batch_start/inacti0µÆ‚ve_split_bytes.all.peak‚ 2671790080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197974763095040 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34248887 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3872283289487 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227866560854016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3874881574355 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34249352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198067123014144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19822767 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203397285667840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69505012 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30603715 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881288378880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30603709 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5330083940864 _step‚169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231463665461760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881288378880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38901303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14426123 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34248887 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3872283289487 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227866560854016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3874881574355 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34249352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198067123014144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19822767 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203397285667840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69505012 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30603715 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3881288378880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30603709 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5330083940864 _step‚169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231463665461760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881288378880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38901303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14426123 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240721197701120 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 _step‚172 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5453013831168 trainer/global_step‚2149 H?DeviceStatsMonitor.on_train_batch_Ì„_é)start/inactive_split.all.peak‚1408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3964532956087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208047100875264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚233077278393344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39818153 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3961934671427 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3971086885888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202594165770240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884393333720942e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31301279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5452963239424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39798121 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237039187636803 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233077252965376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31301279 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3968486677504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31301273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236752737415168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14754336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208047179601408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39798121 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240721197701120 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 _step‚172 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5453013831168 trainer/global_step‚2149 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3964532956087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208047100875264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚233077278393344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39818153 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3961934671427 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3971086885888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202594165770240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884393333720942e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31301279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5452963239424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39798121 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚809 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237039187636803 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233077252965376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31301279 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3968486677504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31301273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236752737415168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14754336¬·ð¬2è- _wandbruntime‚1774  _runtime‚1774*¥û'éýM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20282567 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39836413 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236860677120512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202686553823744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35043532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14760513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236860677120512 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3970319006208 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35043077 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233183619567616 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236860650728960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5455461859328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31315515 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39836413 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5455512437760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237147358153743 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240833596348416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240830969735168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966362298963 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233183594139648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208141987548672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202686525689344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236860650728960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31315509 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20282564 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3970319006208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 trainer/global_step‚2149 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31315509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237149981866579 _step‚173 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240833596348416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864‹å.Y€ê‚N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202686525689344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236860650728960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31315509 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20282564 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3970319006208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 trainer/global_step‚2149 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31315509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237149981866579 _step‚173 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240833596348416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20282567 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39836413 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236860677120512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202686553823744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35043532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14760513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236860677120512 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3970319006208 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35043077 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233183619567616 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236860650728960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5455461859328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚313155  ˆ 15 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39836413 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5455512437760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237147358153743 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240833596348416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240830969735168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966362298963 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233183594139648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208141987548672­lÀiæ2ë- _wandbruntime‚1774  _runtime‚1774„Ÿ»|RìÌ 0loss/SRqqt2/detection_loss‚2.029089417647232 0loss/FRt4/assignment_loss‚3.9281626834588894 %loss/total_loss‚50.09936381440188 epoch‚1 0loss/FBt1/assignment_loss‚0.2421600414782497 /loss/SRqqt3/detection_loss‚2.78837658011395 .loss/FBt2/detection_loss‚2.892120373249054 _step‚174 .loss/FRt1/assignment_loss‚2.56194606423378 0loss/FRt3/assignment_loss‚4.6823150170476815 1loss/SRqqt1/assignment_loss‚1.442880740210618 2loss/SRqqt4/assignment_loss‚1.8778679276315065 /loss/FRt1/detection_loss‚1.6765018395822624 0loss/FRt2/assignment_loss‚4.2072578611828035 0loss/FBt4/assignment_loss‚0.3838802192892347 1loss/SRqqt1/detection_loss‚0.8397379667587477 1loss/SRqqt4/detection_loss‚1.7320823228129973 1loss/SRqqt3/assignment_loss‚2.149329900741577 trainer/global_step‚2149 0loss/SRqqt2/assignment_loss‚1.65154386989095 1loss/FBt3/assignment_loss‚0.35692458660341797 .loss/FRt2/detection_loss‚2.625233401854833 /loss/FRt4/detection_loss‚2.2296595982476775 .loss/FBt1/detection_loss‚0.721059065025964 $ _timestamp‚1.788439334120156e+09 /loss/FBt3/detection_loss‚2.6153895974159242 /loss/FBt4/detection_loss‚3.5792806148529053 _runtime‚1774.164702157 0loss/FBt2/assignment_loss‚0.4929139854386449 .loss/FRt3/detection_loss‚2.393650139632978›WíÑ 0loss/SRqqt2/detection_loss‚2.029089417647232 0loss/FRt4/assignment_loss‚3.9281626834588894 %loss/total_loss‚50.09936381440188 epoch‚1 0loss/FBt1/assignment_loss‚0.2421600414782497 /loss/SRqqt3/detection_loss‚2.78837658011395 .loss/FBt2/detection_loss‚2.892120373249054 _step‚174 .loss/FRt1/assignment_loss‚2.56194606423378 0loss/FRt3/assignment_loss‚4.6823150170476815 1loss/SRqqt1/assignment_loss‚1.442880740210618 2loss/SRqqt4/assignment_loss‚1.8778679276315065 /loss/FRt1/detection_loss‚1.6765018395822624 0loss/FRt2/assignment_loss‚4.2072578611828035 0loss/FBt4/assignment_loss‚0.3838802192892347 1loss/SRqqt1/detection_loss‚0.8397379667587477 1loss/SRqqt4/detection_loss‚1.7320823228129973 1loss/SRqqt3/assignment_loss‚2.149329900741577 trainer/global_step‚2149 0loss/SRqqt2/assignment_loss‚1.65154386989095 1loss/FBt3/assignment_loss‚0.35692458660341797 .loss/FRt2/detection_loss‚2.625233401854833 /loss/FRt4/detection_loss‚2.2296595982476775 .loss/FBt1/detection_loss‚0.721059065025964 $ _timestamp‚1.788439334120156e+09 /loss/FBt3/detection_loss‚2.6153895974159242 /loss/FBt4/detection_loss‚3.5792806148529053 _runtime‚1774.164702157 0loss/FBt2/assignment_loss‚0.4929139854386449 .loss/FRt3/detection_loss‚2.393650139632978®ËuŒ_“î‚(:ˆ °ÖåÔÊ’öÈ gpu.0.gpu‚76.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚238.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Ú…T,ï‚(:¡ °ÖåÔÎøžË# disk.dm-0.out‚16387.78759765625 network.sent‚ 4.119098e+06) disk./.usagePercent‚74.12208001902566 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 75.25390625& proc.memory.availableMB‚ 5.83203125 disk.rbd3.in‚0& disk.nvme0n1.out‚16387.78759765625 disk.rbd0.in‚0 disk.dm-0.in‚ 1724.57421875 disk.dm-1.out‚ 75.25390625 disk.sda2.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.57421875# disk.nvme0c0n1.in‚ 1724.57421875 disk.drbd1777.out‚0! disk.rbd2.out‚101.88232421875# proc.memory.rssMB‚ 2760.99609375 network.recv‚693175 disk.sda1.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚16387.78759765625$ memory_percent‚99.96440410614014 proc.cpu.threads‚17$ disk./.usageGB‚1104.472557067871 disk.sda.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.85178279876709 disk.sda1.out‚0 disk.rbd1.out‚ 0.85693359375 disk.rbd2.in‚0 disk.sda3.out‚ 75.25390625 disk.rbd0.out‚0 disk.dm-1.in‚0 cpu‚25.197540616315063çKë“ð‚(:ˆ ¿ÖåÔöžáÈ gpu.0.gpu‚89.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄŽPŒñ‚(: ¿ÖåÔñ¡Ì) disk./.usagePercent‚74.12208001902566 disk.rbd3.in‚0% disk.nvme0n1.out‚16486.9697265625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1724.57421875 network.sent‚ 4.228273e+06 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚76.125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚16486.9697265625 disk.sda1.out‚0$ memory_percent‚99.96592998504639& proc.memory.availableMB‚ 5.58203125 disk.rbd3.out‚0 disk.dm-0.in‚ 1724.57421875! disk.rbd2.out‚101.88232421875 disk.sda.out‚76.125 disk.sda1.in‚0 cpu‚25.196300166938226 disk.rbd0.in‚0' disk.nvme0c0n1.out‚16486.9697265625 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.85693359375# proc.memory.rssMB‚ 2760.99609375 network.recv‚699901$ disk./.usageGB‚1104.472557067871 disk.dm-1.out‚76.125 disk.rbd1.in‚0) proc.memory.percent‚16.85178279876709! disk.nvme0n1.in‚ 1724.57421875 disk.dm-1.in‚0Á›ˆJ2ò- _wandbruntime‚1812  _runtime‚1812`Ï ¹Éóà _step‚175 ' lr-AdamW/pg1‚0.00037965267192818607 ' lr-AdamW/pg2‚0.00037965267192818607 trainer/global_step‚2199 % _timestamp‚1.7884393728195803e+09 _runtime‚1812.864243855“ËŠÎôÈ _runtime‚1812.864243855 _step‚175 ' lr-AdamW/pg1‚0.00037965267192818607 ' lr-AdamW/pg2‚0.00037965267192818607 trainer/global_step‚2199 % _timestamp‚1.7884393728195803e+09¯:ZâP2õ- _wandbruntime‚1812  _runtime‚1812\£X/ú'öôO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40733205 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238394311678976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207213540311040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212792118535680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 _step‚176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4062717734912 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246209813817344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35822592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5578550090240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242447727075011 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32013079 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40713173 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242149696290816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242149722682368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4060117526528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40713173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20733171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238394337106944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32013073 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15089421 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207213568445440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32013073 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15089421 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207213568445440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207213540311040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2671790080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212792118535680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 _step‚176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4062717734912 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246209813817344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35822592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5578550090240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚242447727075011 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32013079 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40713173 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242149696290816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242149722682368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4060117526528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40713173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20733171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238394337106944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552°h¥Î<2ø- _wandbruntime‚1813  _runtime‚1813}»=œ› ùþM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242558521304787 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1813.608073127 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4064550076928 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4064550076928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5581059326976 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _step‚177 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32027315 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246322212464640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35837943 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242257635996160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20742367 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238500678281216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242257662387712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40751465 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35838415 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40731407 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246319585851392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20742364 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246322212464640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32027315 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4061949855232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5581008748544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40751465 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207305956498944 trainer/global_step‚2199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242257662387712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207305928364544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15096048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15095579 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40731407 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32027309 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212886937113088ÝsÈþ 'úƒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4064550076928 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2671790080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4064550076928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5581059326976 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _step‚177 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32027315 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246322212464640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 354333184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20058 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35837943 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242257635996160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20742367 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238500678281216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242257662387712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40751465 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35838415 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40731407 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246319585851392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20742364 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246322212464640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32027315 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4061949855232 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5581008748544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40751465 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207305956498944 trainer/global_step‚2199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚809 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242257662387712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207305928364544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15096048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15095579 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40731407 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32027309 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212886937113088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚242558521304787 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1813.608073127 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0±ÒWv2û- _wandbruntime‚1813  _runtime‚1813ÿöªÆYüÓ 1loss/SRqqt2/detection_loss‚1.6258484630261438 _step‚178 trainer/global_step‚2199 .loss/FRt1/detection_loss‚1.586461606171896 /loss/FBt2/detection_loss‚3.4109557271003723 0loss/FBt3/assignment_loss‚0.5409597804246005 1loss/FBt1/assignment_loss‚0.34558596781337503 .loss/FRt3/detection_loss‚3.026546957992738 /loss/FRt4/detection_loss‚2.5044500452280043 % _timestamp‚1.7884393735650492e+09 /loss/FRt2/detection_loss‚2.2900490456936407 /loss/FRt3/assignment_loss‚5.080945976318851 2loss/SRqqt1/assignment_loss‚1.4041857629199512 2loss/SRqqt3/assignment_loss‚2.3177151427124487 /loss/FBt1/detection_loss‚0.6922019990000365 epoch‚1 2loss/SRqqt4/assignment_loss‚1.8394072300151867 1loss/SRqqt1/detection_loss‚1.1170797648373991 .loss/FBt3/detection_loss‚2.757049947977066 /loss/FBt4/detection_loss‚2.7581049295572133 &loss/total_loss‚50.837431852403974 _runtime‚1813.609638737 1loss/SRqqt2/assignment_loss‚2.154258088666504 1loss/FBt2/assignment_loss‚0.35961621541840333 0loss/SRqqt3/detection_loss‚2.553933644836599 1loss/SRqqt4/detection_loss‚2.1551839183358585 /loss/FRt4/assignment_loss‚3.883821203708649 /loss/FBt4/assignment_loss‚0.556430531020921 0loss/FRt1/assignment_loss‚2.2292819056870803 /loss/FRt2/assignment_loss‚3.647357997941036>k^ýØ /loss/FRt2/detection_loss‚2.2900490456936407 /loss/FRt3/assignment_loss‚5.080945976318851 2loss/SRqqt1/assignment_loss‚1.4041857629199512 2loss/SRqqt3/assignment_loss‚2.3177151427124487 /loss/FBt1/detection_loss‚0.6922019990000365 epoch‚1 2loss/SRqqt4/assignment_loss‚1.8394072300151867 1loss/SRqqt1/detection_loss‚1.1170797648373991 .loss/FBt3/detection_loss‚2.757049947977066 /loss/FBt4/detection_loss‚2.7581049295572133 &loss/total_loss‚50.837431852403974 _runtime‚1813.609638737 1loss/SRqqt2/assignment_loss‚2.154258088666504 1loss/FBt2/assignment_loss‚0.35961621541840333 0loss/SRqqt3/detection_loss‚2.553933644836599 1loss/SRqqt4/detection_loss‚2.1551839183358585 /loss/FRt4/assignment_loss‚3.883821203708649 /loss/FBt4/assignment_loss‚0.556430531020921 0loss/FRt1/assignment_loss‚2.2292819056870803 /loss/FRt2/assignment_loss‚3.647357997941036 1loss/SRqqt2/detection_loss‚1.6258484630261438 _step‚178 trainer/global_step‚2199 .loss/FRt1/detection_loss‚1.586461606171896 /loss/FBt2/detection_loss‚3.4109557271003723 0loss/FBt3/assignment_loss‚0.5409597804246005 1loss/FBt1/assignment_loss‚0.34558596781337503 .loss/FRt3/detection_loss‚3.026546957992738 /loss/FRt4/detection_loss‚2.5044500452280043 % _timestamp‚1.7884393735650492e+09²ijYw“þ‚(:ˆ ÎÖåÔõ¼É gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚230.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.0574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…#ÿ‚(:˜ ÎÖåÔâìÓÌ disk.rbd2.in‚0! disk.nvme0n1.in‚ 1724.57421875& disk.nvme0c0n1.out‚16586.638671875 network.sent‚ 4.233463e+06) disk./.usagePercent‚74.12208001902566 disk.sda3.out‚ 76.72265625 network.recv‚703413 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1724.57421875% proc.memory.availableMB‚ 5.6015625$ disk.nvme0n1.out‚16586.638671875 disk.rbd0.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.919921875! disk.rbd2.out‚103.15673828125 disk.dm-0.in‚ 1724.57421875 disk.drbd1777.out‚0 disk.dm-1.out‚ 76.72265625$ memory_percent‚99.96581077575684$ disk./.usageGB‚1104.472557067871 disk.sda.out‚ 76.72265625 disk.sda3.in‚0# proc.memory.rssMB‚ 2760.99609375 cpu‚25.194961794810222! disk.dm-0.out‚16586.638671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0) proc.memory.percent‚16.85178279876709@™™’€‚(:‡ ÝÖåÔû¬Ê gpu.0.gpu‚70.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.201 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷“!‚(:‹ ÝÖåÔƧ¥Ë! disk.dm-0.out‚16586.974609375! disk.nvme0n1.in‚ 1724.57421875 disk.dm-0.in‚ 1724.57421875! disk.rbd2.out‚104.23095703125 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 77.5234375 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1724.57421875 proc.cpu.threads‚17 network.sent‚ 4.342175e+06$ disk./.usageGB‚1104.472557067871 disk.rbd1.in‚0$ memory_percent‚99.96426105499268) disk./.usagePercent‚74.12208001902566 disk.rbd3.out‚0$ disk.nvme0n1.out‚16586.974609375 disk.rbd0.out‚0 network.recv‚709875 disk.dm-1.out‚ 77.5234375 disk.sda.out‚ 77.5234375& disk.nvme0c0n1.out‚16586.974609375' proc.memory.percent‚16.851806640625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 5.85546875 proc.memory.rssMB‚2761 cpu‚25.193829091786277 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0 disk.sda2.out‚0rþÇ’‚‚(:‡ ìÖåÔŒÂªÉ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.427 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]P郂(:… ìÖåÔò´¿Ë disk.dm-0.in‚ 1724.57421875$ memory_percent‚99.96469020843506! disk.rbd2.out‚105.41845703125 disk.dm-1.in‚0 disk.sda.out‚ 77.9765625" disk.nvme0n1.out‚ 16686.2421875# disk.nvme0c0n1.in‚ 1724.57421875 network.recv‚714100 disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.12208001902566 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 network.sent‚ 4.348113e+06 disk.sda.in‚0! disk.nvme0n1.in‚ 1724.57421875 disk.dm-0.out‚ 16686.2421875' proc.memory.percent‚16.851806640625 cpu‚25.192537563614025$ disk./.usageGB‚1104.472557067871 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 77.9765625 proc.memory.rssMB‚2761 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 16686.2421875& proc.memory.availableMB‚ 5.78515625 disk.dm-1.out‚ 77.9765625 disk.sda2.out‚0 disk.rbd0.out‚0Šå?2„- _wandbruntime‚1852  _runtime‚1852(—œÇ…Á % _timestamp‚1.7884394122607589e+09 _runtime‚1852.305523349 _step‚179 & lr-AdamW/pg1‚0.0003788446599822273 & lr-AdamW/pg2‚0.0003788446599822273 trainer/global_step‚2249¡Œ‚Ì†Æ & lr-AdamW/pg1‚0.0003788446599822273 & lr-AdamW/pg2‚0.0003788446599822273 trainer/global_step‚2249 % _timestamp‚1.7884394122607589e+09 _runtime‚1852.305523349 _step‚179³äý|u2‡- _wandbruntime‚1852  _runtime‚1852\€ è>ˆôO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32724879 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21192971 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15423627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154348583936 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚810 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217536643615232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚1852.306617701 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247546681558016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154348583936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628225 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15423144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211832971120640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4151748375552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217536562791936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52688896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5703619805696 % _timestamp‚1.7884394122616215e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32724873 trainer/global_step‚2249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21192968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251701056533504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247546707949568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2678081536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247858890225847 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251698429933568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247856266513219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41648257 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251698429933568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1409 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211832942986240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36616598 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243711395820544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32724873 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41648257 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5703672494592 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚483 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32724879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 360215552èsmÿ'‰ùO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚483 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32724879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 360215552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32724879 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21192971 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15423627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154348583936 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74373136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚810 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217536643615232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚1852.306617701 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247546681558016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154348583936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628225 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15423144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211832971120640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4151748375552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217536562791936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52688896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5703619805696 % _timestamp‚1.7884394122616215e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32724873 trainer/global_step‚2249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21192968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251701056533504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247546707949568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2678081536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247858890225847 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251698429933568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247856266513219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41648257 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251698429933568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1409 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211832942986240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36616598 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243711395820544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32724873 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41648257 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5703672494592 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122´¶P2Š- _wandbruntime‚1853  _runtime‚1853…%'‹ýM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 360215552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74385568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41646459 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41666517 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21202167 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211925331039744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251810828580864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788439413004108e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15430180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247964437030159 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156180925952 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚243817736994816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74405632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32739115 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36631885 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32739109 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4153580704256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2249 _runtime‚1853.049497243 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36632347 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156180925952 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚243817736994816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74405632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32739115 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36631885 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32739109 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4153580704256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2249 _runtime‚1853.049497243 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36632347 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15430180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247964437030159 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257187046049792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5829132224512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37411366 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222281427294208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33436673 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222281506020352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2678081536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2299 % _timestamp‚1.7884394515591857e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257189672649728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15758598 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 360215552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252943666825216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4238975129911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1409 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚810 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42543277 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42543277 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216452373795840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257187046049792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252943693216768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253267429664055nÿ'›ùO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50591744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚810 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42543277 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42543277 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216452373795840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257187046049792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252943693216768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253267429664055 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37411838 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚216452345661440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21652768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5829081632768 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _runtime‚1891.604064647 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249028454534144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252943666825216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257187046049792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5829132224512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78726144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37411366 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222281427294208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33436673 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222281506020352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2678081536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2299 % _timestamp‚1.7884394515591857e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257189672649728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15758598 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 360215552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252943666825216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4238975129911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1409 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3¸2y2œ- _wandbruntime‚1892  _runtime‚1892“þ¾'KþM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33450915 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4245211553280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4245211553280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33450915 trainer/global_step‚2299 _step‚185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21661964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257299444697088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4247811774976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257296818083840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216544761849344 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76012420 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253051632922112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222376411347968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42581569 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.O楹peak‚1573 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42561511 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _runtime‚1892.349452441 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2678081536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253375600181203 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 % _timestamp‚1.7884394523041005e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33450909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37427166 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 360215552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253051606530560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4247811774976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661967 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253051606530560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33450909 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42561511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224r_‹± 'žƒN cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216544761849344 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76012420 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253051632922112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222376411347968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42581569 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42561511 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _runtime‚1892.349452441 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2678081536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48481280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253375600181203 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 % _timestamp‚1.7884394523041005e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33450909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37427166 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 360215552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253051606530560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4247811774976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661967 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253051606530560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33450909 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42561511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33450915 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4245211553280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4245211553280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33450915 trainer/global_step‚2299 _step‚185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21661964 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257299444697088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4247811774976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257296818083840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753¹Fš×2Ÿ- _wandbruntime‚1892  _runtime‚1892PŠmdW Ñ .loss/FRt4/detection_loss‚2.338290494349268 0loss/SRqqt3/detection_loss‚2.447475865483284 /loss/FRt2/assignment_loss‚3.947145678559128 /loss/FRt2/detection_loss‚2.3823236275692374 .loss/FBt3/detection_loss‚3.998482902844747 /loss/FBt4/detection_loss‚2.6773489117622375 _runtime‚1892.350622837 _step‚186 2loss/SRqqt1/assignment_loss‚1.3619609001572386 2loss/SRqqt4/assignment_loss‚1.8857588626836475 1loss/FBt1/assignment_loss‚0.15603980679795285 0loss/FBt4/assignment_loss‚0.3608566423257192 epoch‚1 /loss/FRt4/assignment_loss‚3.762253631044317 0loss/SRqqt3/assignment_loss‚2.66828174575379 0loss/FBt2/assignment_loss‚0.3027780871440579 /loss/FRt1/detection_loss‚1.6672362140738046 % _timestamp‚1.7884394523060095e+09 /loss/FBt1/detection_loss‚0.5179405398618857 trainer/global_step‚2299 2loss/FBt3/assignment_loss‚0.053923097594330706 /loss/FRt3/detection_loss‚2.5653671270761733 1loss/SRqqt1/detection_loss‚1.0839430540800095 &loss/total_loss‚49.475488726459304 /loss/FRt1/assignment_loss‚1.906181343472921 1loss/SRqqt2/assignment_loss‚1.914157380876334 1loss/SRqqt2/detection_loss‚2.0780871711347415 .loss/FRt3/assignment_loss‚4.92653550245823 1loss/SRqqt4/detection_loss‚1.8300813995955283 .loss/FBt2/detection_loss‚2.643038739760717ž×\h\¡Ö 2loss/FBt3/assignment_loss‚0.053923097594330706 /loss/FRt3/detection_loss‚2.5653671270761733 1loss/SRqqt1/detection_loss‚1.0839430540800095 &loss/total_loss‚49.475488726459304 /loss/FRt1/assignment_loss‚1.906181343472921 1loss/SRqqt2/assignment_loss‚1.914157380876334 1loss/SRqqt2/detection_loss‚2.0780871711347415 .loss/FRt3/assignment_loss‚4.92653550245823 1loss/SRqqt4/detection_loss‚1.8300813995955283 .loss/FBt2/detection_loss‚2.643038739760717 .loss/FRt4/detection_loss‚2.338290494349268 0loss/SRqqt3/detection_loss‚2.447475865483284 /loss/FRt2/assignment_loss‚3.947145678559128 /loss/FRt2/detection_loss‚2.3823236275692374 .loss/FBt3/detection_loss‚3.998482902844747 /loss/FBt4/detection_loss‚2.6773489117622375 _runtime‚1892.350622837 _step‚186 2loss/SRqqt1/assignment_loss‚1.3619609001572386 2loss/SRqqt4/assignment_loss‚1.8857588626836475 1loss/FBt1/assignment_loss‚0.15603980679795285 0loss/FBt4/assignment_loss‚0.3608566423257192 epoch‚1 /loss/FRt4/assignment_loss‚3.762253631044317 0loss/SRqqt3/assignment_loss‚2.66828174575379 0loss/FBt2/assignment_loss‚0.3027780871440579 /loss/FRt1/detection_loss‚1.6672362140738046 % _timestamp‚1.7884394523060095e+09 /loss/FBt1/detection_loss‚0.5179405398618857 trainer/global_step‚2299ºqG1ž¢‚(:“ ¨×åÔŽìˆÉ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.962799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž¿eÆ£‚(:„ ¨×åÔÓàéË disk.dm-1.out‚ 80.2578125 disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 8.76953125 disk.sda.out‚ 80.2578125) disk./.usagePercent‚74.12214709299964 disk.drbd1777.in‚0! disk.rbd2.out‚107.84814453125 disk.sda2.in‚0' proc.memory.percent‚16.851806640625 cpu‚25.18810176291705 network.sent‚ 4.473835e+06 disk.sda3.out‚ 80.2578125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 proc.memory.rssMB‚2761$ disk.nvme0c0n1.out‚ 16984.2578125 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1724.57421875 disk.rbd0.out‚0 disk.drbd1777.out‚0# memory_percent‚99.9464750289917 network.recv‚732859" disk.nvme0n1.out‚ 16984.2578125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1724.57421875 disk.dm-0.out‚ 16984.2578125 disk.sda.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1104.4735565185547 disk.rbd0.in‚0.¡.dž¤‚(:“ ·×åÔÌÿùÈ gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m½ä¥‚(:ˆ ·×åÔ‘ï™Ì disk.sda.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0" disk.nvme0n1.out‚ 17083.3984375# memory_percent‚99.9464750289917& proc.memory.availableMB‚ 8.76953125 cpu‚25.186899332539674 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.919921875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 17083.3984375' proc.memory.percent‚16.851806640625 network.sent‚ 4.582921e+06% disk./.usageGB‚1104.4735374450684 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚2761 disk.dm-1.out‚ 81.06640625 disk.sda.out‚ 81.06640625 disk.sda3.out‚ 81.06640625! disk.nvme0n1.in‚ 1724.57421875) disk./.usagePercent‚74.12214581296196 disk.dm-0.out‚ 17083.3984375 disk.rbd2.in‚0 network.recv‚739439 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.57421875 disk.drbd1777.in‚0! disk.rbd2.out‚108.92236328125 proc.cpu.threads‚17 disk.rbd1.in‚0[×g‚2¦- _wandbruntime‚1930  _runtime‚1930>NrǧÁ & lr-AdamW/pg1‚0.0003771130965737591 & lr-AdamW/pg2‚0.0003771130965737591 trainer/global_step‚2349 % _timestamp‚1.7884394909120169e+09 _runtime‚1930.956744987 _step‚187‰¤ÄÏÌ¨Æ trainer/global_step‚2349 % _timestamp‚1.7884394909120169e+09 _runtime‚1930.956744987 _step‚187 & lr-AdamW/pg1‚0.0003771130965737591 & lr-AdamW/pg2‚0.0003771130965737591»¾{ê92©- _wandbruntime‚1930  _runtime‚1930`~¿!ú'ªôO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43478361 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16093772 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚1930.957885725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2678081536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16093291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4335010073600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258340652092416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34148479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38205859 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4330455854519 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43478361 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚810 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4337610281984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5954728700928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258673345389635 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254345487819776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34148473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258340652092416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258340678483968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚484 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22112571 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262675662166016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 % _timestamp‚1.7884394909128196e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262678288765952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5954779292672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258675969102263 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34148473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227026555763712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221071776471040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 360215552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262675662166016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4327857569859 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43458329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262678288765952Ún±B¡«ùO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5954728700928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258673345389635 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254345487819776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34148473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25ëŽl^%8340652092416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258340678483968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚484 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22112571 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262675662166016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630560 % _timestamp‚1.7884394909128196e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262678288765952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5954779292672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258675969102263 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34148473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227026555763712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221071776471040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 360215552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262675662166016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4327857569859 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43458329 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262678288765952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340224 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43478361 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16093772 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚1930.957885725 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2678081536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16093291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4335010073600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258340652092416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34148479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38205859 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4330455854519 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43478361 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚810 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4337610281984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552¼òé!ô2¬- _wandbruntime‚1931  _runtime‚1931ÃÄ´W'­þM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4332285197395 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4339442624000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38221702 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262788060813312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4339442624000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262785434200064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1409 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258448618189312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43476563 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227121359114240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258448591797760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329686912527 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5957273302528 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43496621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16099938 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258448618189312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1931.699786458 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4336842402304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22121767 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2678081536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5957222724096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258781515906575 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34162715 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0µ¹U '®ƒN _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221164136390144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50578432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚254451828994048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38221702 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262788060813312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4339442624000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262785434200064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1409 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258448618189312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43476563 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227121359114240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258448591797760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329686912527 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5957273302528 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43496621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16099938 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258448618189312 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78712832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1931.699786458 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4336842402304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22121767 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2678081536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5957222724096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258781515906575 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77659336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34162715 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4332285197395 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4339442624000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Eeœ(·‚(: ä×åÔ˜™îË disk.drbd1777.out‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 1724.57421875 proc.cpu.threads‚17 network.recv‚754112 disk.sda3.in‚0' disk.nvme0c0n1.out‚17282.5732421875) proc.memory.percent‚16.85183048248291 disk.drbd1777.in‚0 disk.rbd1.in‚0! disk.rbd2.out‚111.37255859375 disk.sda2.in‚0% disk.nvme0n1.out‚17282.5732421875& proc.memory.availableMB‚ 8.76171875 disk.rbd1.out‚ 0.919921875 disk.sda2.out‚0 disk.sda3.out‚ 82.84765625 disk.rbd2.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2761.00390625 cpu‚25.183690016667004 disk.dm-1.in‚0 disk.dm-1.out‚ 82.84765625 disk.sda.out‚ 82.84765625 disk.rbd3.out‚0) disk./.usagePercent‚74.12213454863046 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚99.94652271270752 network.sent‚ 4.702588e+06 disk.dm-0.in‚ 1724.57421875 disk.sda1.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1724.57421875% disk./.usageGB‚1104.4733695983887" disk.dm-0.out‚17282.5732421875oI˜X2¸- _wandbruntime‚1970  _runtime‚1970ÚCFéƹÀ % _timestamp‚1.7884395301089044e+09 _runtime‚ 1970.15371483 _step‚191 & lr-AdamW/pg1‚0.0003761899059664686 & lr-AdamW/pg2‚0.0003761899059664686 trainer/global_step‚2399WN®#˺Š& lr-AdamW/pg2‚0.0003761899059664686 trainer/global_step‚2399 % _timestamp‚1.7884395301089044e+09 _runtime‚ 1970.15371483 _step‚191 & lr-AdamW/pg1‚0.0003761899059664686¿>2»- _wandbruntime‚1970  _runtime‚1970£¥9Èù'¼óO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4426640922624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268164278282240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44393413 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚264081884827843 $ _timestamp‚1.788439530109742e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231770844850176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44393413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39001889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264084508540471 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34860273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22572371 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259662571961344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34860279 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263737663751168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16429027 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231770925673472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚268166904882176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚1970.154686871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4429241131008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44373381 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22572368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6079746527232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419338294467 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚494 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34860279 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34860273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16429518 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2680178688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52688896 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4421936579127 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24122 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4426640922624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712628 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268164278282240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44393413 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20032 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚264081884827843 $ _timestamp‚1.788439530109742e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626599936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231770844850176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44393413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39001889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465225 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264084508540471 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34860273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22572371 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259662571961344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34860279 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263737663751168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16429027 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626599936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231770925673472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚268166904882176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚1970.154686871 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4429241131008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44373381 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24122 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22572368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6079746527232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419338294467 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚494 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34860279 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34860273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16429518 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2680178688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52688896 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284660 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4421936579127 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4428473251328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22581567 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6082195219968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34874515 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225783567199744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44411673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚268276676929536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231865734285312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34874509 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225783539065344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4431073473024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 _step‚193 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 trainer/global_step‚2399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34874515 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259768913135616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231865815095296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259768887707648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264192679057619 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4431073473024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4423765922003 _runtime‚1970.894951341 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44391615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44411673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/numîÛ°ål _oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263845577064960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80809984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39017153 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22581564 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚268274050316288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264190055344783^0 'ÀƒN T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚268276676929536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39017623 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884395308496892e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263845577064960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44391615 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52675584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4428473251328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22581567 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6082195219968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712836 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34874515 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225783567199744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20058 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24122 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284868 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44411673 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚268276676929536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231865734285312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34874509 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225783539065344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4431073473024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600221696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247568896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 _step‚193 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465225 trainer/global_step‚2399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34874515 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259768913135616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231865815095296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259768887707648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264192679057619 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4431073473024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4423765922003 _runtime‚1970.894951341 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44391615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44411673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263845577064960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80809984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600221696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39017153 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22581564 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚268274050316288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264190055344783Á“ú 2Á- _wandbruntime‚1970  _runtime‚1970tHSYÂÓ /loss/FRt3/assignment_loss‚4.929031280370859 /loss/FRt3/detection_loss‚3.1393078290499172 /loss/FBt1/detection_loss‚0.6537055859395022 %loss/total_loss‚50.18908242502236 1loss/FBt3/assignment_loss‚0.27302900794893503 0loss/FBt4/assignment_loss‚0.4477714426604507 /loss/FRt1/detection_loss‚1.8064804213088856 epoch‚1 0loss/FRt2/assignment_loss‚3.4068184375762938 .loss/FRt4/assignment_loss‚3.89790415763855 1loss/SRqqt2/assignment_loss‚2.053220509521423 /loss/FRt4/detection_loss‚2.6272061804407523 1loss/SRqqt2/detection_loss‚1.7024454464835506 1loss/SRqqt3/detection_loss‚2.7757687962055204 trainer/global_step‚2399 _step‚194 % _timestamp‚1.7884395308518507e+09 2loss/SRqqt4/assignment_loss‚1.9457745475901498 0loss/FBt2/assignment_loss‚0.3586710775271058 /loss/FRt1/assignment_loss‚2.078707106280745 0loss/FBt1/assignment_loss‚0.2176412161148279 .loss/FRt2/detection_loss‚2.593603068590164 _runtime‚1970.896449564 1loss/SRqqt1/detection_loss‚1.0257389940894568 1loss/SRqqt4/detection_loss‚2.1622667140430876 /loss/FBt2/detection_loss‚3.0192063570022585 /loss/FBt3/detection_loss‚2.5972976854869296 2loss/SRqqt1/assignment_loss‚1.2763179509112468 1loss/SRqqt3/assignment_loss‚2.493792161941528 /loss/FBt4/detection_loss‚2.7073764503002167KHšþ^ÃØ 1loss/FBt3/assignment_loss‚0.27302900794893503 0loss/FBt4/assignment_loss‚0.4477714426604507 /loss/FRt1/detection_loss‚1.8064804213088856 epoch‚1 0loss/FRt2/assignment_loss‚3.4068184375762938 .loss/FRt4/assignment_loss‚3.89790415763855 1loss/SRqqt2/assignment_loss‚2.053220509521423 /loss/FRt4/detection_loss‚2.6272061804407523 1loss/SRqqt2/detection_loss‚1.7024454464835506 1loss/SRqqt3/detection_loss‚2.7757687962055204 trainer/global_step‚2399 _step‚194 % _timestamp‚1.7884395308518507e+09 2loss/SRqqt4/assignment_loss‚1.9457745475901498 0loss/FBt2/assignment_loss‚0.3586710775271058 /loss/FRt1/assignment_loss‚2.078707106280745 0loss/FBt1/assignment_loss‚0.2176412161148279 .loss/FRt2/detection_loss‚2.593603068590164 _runtime‚1970.896449564 1loss/SRqqt1/detection_loss‚1.0257389940894568 1loss/SRqqt4/detection_loss‚2.1622667140430876 /loss/FBt2/detection_loss‚3.0192063570022585 /loss/FBt3/detection_loss‚2.5972976854869296 2loss/SRqqt1/assignment_loss‚1.2763179509112468 1loss/SRqqt3/assignment_loss‚2.493792161941528 /loss/FBt4/detection_loss‚2.7073764503002167 /loss/FRt3/assignment_loss‚4.929031280370859 /loss/FRt3/detection_loss‚3.1393078290499172 /loss/FBt1/detection_loss‚0.6537055859395022 %loss/total_loss‚50.18908242502236Âö¾£ªÄ‚(:’ ó×åÔãâ¯É gpu.0.gpu‚73.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚234.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚46.97559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€¦²"Å‚(:— ó×åÔ–±šÍ network.recv‚758737) disk./.usagePercent‚74.12222415126743 disk.sda3.in‚0! disk.dm-0.out‚17383.166015625 cpu‚25.182655234818213% disk./.usageGB‚1104.4747047424316 disk.sda3.out‚ 83.5703125 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1724.57421875& proc.memory.availableMB‚ 8.72265625# proc.memory.rssMB‚ 2761.00390625 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1724.57421875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚17383.166015625$ memory_percent‚99.94676113128662 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd1.out‚ 0.919921875! disk.rbd2.out‚112.38427734375 network.sent‚ 4.708774e+06 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚17383.166015625# disk.nvme0c0n1.in‚ 1724.57421875) proc.memory.percent‚16.85183048248291 disk.sda2.in‚0 disk.dm-1.out‚ 83.5703125 disk.rbd2.in‚0 disk.sda.out‚ 83.5703125ÉŽ] ’Æ‚(:‡ ‚ØåÔ“§ŠÉ gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚42.0 gpu.0.powerWatts‚95.399' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hz‡"Ç‚(:— ‚ØåÔÞ¥ýÊ) disk./.usagePercent‚74.12218754219005 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 84.015625 proc.cpu.threads‚17 network.sent‚ 4.816767e+06 disk.rbd3.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 network.recv‚764348 disk.sda3.out‚ 84.015625 disk.dm-0.in‚ 1724.57421875# disk.dm-0.out‚17482.89111328125 disk.drbd1777.in‚0 disk.rbd2.out‚ 113.638671875 disk.sda.out‚ 84.015625# proc.memory.rssMB‚ 2786.85546875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0$ memory_percent‚99.97491836547852& disk.nvme0n1.out‚17482.89111328125# disk.nvme0c0n1.in‚ 1724.57421875 disk.sda.in‚0 cpu‚25.205716228632088! disk.nvme0n1.in‚ 1724.57421875( disk.nvme0c0n1.out‚17482.89111328125 disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0* proc.memory.percent‚17.009615898132324 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1104.4741592407227 disk.rbd3.out‚0$ proc.memory.availableMB‚4.109375ÈíUaÈ‚(:… ‘ØåÔíùéÈ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚36.0 gpu.0.powerWatts‚89.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â(œ$É‚(:™ ‘ØåÔŠŠ¯Í! disk.nvme0n1.in‚ 1724.57421875 disk.sda3.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 cpu‚25.239610434124533 disk.sda2.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-1.out‚ 84.65234375 network.recv‚768302% disk./.usageGB‚1104.4741592407227 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 84.65234375& proc.memory.availableMB‚ 4.25390625 network.sent‚ 4.822635e+06 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚17483.185546875$ memory_percent‚99.97403621673584* proc.memory.percent‚17.009615898132324 disk.sda.out‚ 84.65234375 disk.rbd2.out‚ 113.701171875 disk.sda.in‚0! disk.dm-0.out‚17483.185546875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.dm-0.in‚ 1724.57421875# proc.memory.rssMB‚ 2786.85546875) disk./.usagePercent‚74.12218754219005 disk.rbd3.in‚0& disk.nvme0c0n1.out‚17483.185546875 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875M”#ГÊ‚(:ˆ  ØåÔÿà·É gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Ì”%Ë‚(:š  ØåÔùƶË disk.drbd1777.out‚0! disk.rbd2.out‚114.89306640625 proc.cpu.threads‚17) disk./.usagePercent‚74.12218754219005 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1724.57421875" disk.dm-0.out‚17582.5517578125* proc.memory.percent‚16.678428649902344 disk.sda.out‚ 85.43359375 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 85.43359375 disk.rbd2.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚17582.5517578125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚17582.5517578125& proc.memory.availableMB‚ 58.5390625 cpu‚25.272753948158893 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.57421875 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2732.59375 network.sent‚ 4.830655e+06 network.recv‚772468% disk./.usageGB‚1104.4741592407227 disk.sda1.in‚0 disk.sda3.out‚ 85.43359375 disk.rbd1.in‚0# memory_percent‚99.6427059173584! disk.nvme0n1.in‚ 1724.57421875Òð‰«2Ì- _wandbruntime‚2036  _runtime‚2036Óˉ”c*ÍÝT TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975011840 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1411 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 106588672 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚269591010233367 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚23135665 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚265113084880896 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚273888022480384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2647 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4487308532224 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35664564 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚119 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚273889997492224 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15972 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1573 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80741807 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚236634703824896 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1074 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚35664569 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80741807 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚236634579587584 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630560 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 20369637376 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚269402688960000 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4485357814784 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚273889997492224 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4485357814784 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚39897594 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80757784 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15972 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950717440 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975011840 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚45077243 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1573 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24122 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚265113108211712 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4477925352471 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24122 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2686470144 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚269592982366823 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23668457472 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1573 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚230483669881344 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2647 _step‚195 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45093215 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133456 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23668457472 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚117 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465225 % _timestamp‚1.7884395963488185e+09 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚35664564 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15977 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚35664569 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚16761929 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚230483687529984 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2417 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788340224 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950717440 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚16761812 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 124237312 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45093215 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788340224 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6150909706240 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247568896 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2647 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚45077243iì.ܾÎâT VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚39897594 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80757784 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15972 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950717440 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 WHDeviceStatsMŽ¿ª'onitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975011840 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚45077243 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1573 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24122 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚265113108211712 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4477925352471 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24122 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2686470144 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚269592982366823 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23668457472 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1573 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚230483669881344 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2647 _step‚195 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45093215 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133456 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23668457472 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚117 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465225 % _timestamp‚1.7884395963488185e+09 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚35664564 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15977 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚35664569 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚16761929 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚230483687529984 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2417 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788340224 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950717440 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚16761812 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 124237312 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45093215 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788340224 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6150909706240 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247568896 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2647 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚45077243 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975011840 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1411 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 106588672 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚269591010233367 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚23135665 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚265113084880896 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚273888022480384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2647 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4487308532224 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35664564 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚119 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚273889997492224 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15972 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1573 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80741807 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚236634703824896 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1074 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚35664569 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80741807 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚236634579587584 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630560 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 20369637376 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚269402688960000 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4485357814784 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚273889997492224 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4485357814784 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247568896 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376ÃvŒpó2Ï- _wandbruntime‚2037  _runtime‚2037h6m)ÐçR SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚35675463 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚269476447547904 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 1975011840 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6151383909376 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950717440 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24122 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚273962255271424 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚230545652235776 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788340224 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247568896 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚16766050 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚236697160382464 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _step‚196 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4480346840999 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3298820096 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 _runtime‚2037.364091444 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 106588672 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚269476423253504 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975011840 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35675458 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23668457472 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1573 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4485832017920 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45084457 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4487782735360 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15972 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚273964230283264 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚269661484020055 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚23143595 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚273962255271424 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15972 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚16766167 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚230545669884416 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚119 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚39909643 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚117 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 124237312 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 trainer/global_step‚2417 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23668457472 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950717440 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133456 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4487782735360 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1411 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6151490498048 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884395973190222e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚236697036145152 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247568896 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2647 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35675458 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚812 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚265183085981696 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2647 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24122 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚273964230283264 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚269663456153511 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630560 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚45100429 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35675463 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4478398038359 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2647 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚119 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚39909643 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚117 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 124237312 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 trainer/global_step‚2417 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23668457472 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950717440 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133456 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4487782735360 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1411 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6151490498048 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884395973190222e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚236697036145152 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247568896 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2647 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35675458 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚812 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚265183085981696 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2647 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24122 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚273964230283264 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚269663456153511 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630560 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚45100429 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35675463 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4478398038359 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2647 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04325397312641144 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_7_of_7‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19532573223114014 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29925599694252014 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.7639520168304443 &particle/accuracy‚0.84184330701828 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4089580178260803 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3476034700870514 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48301976919174194 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12278897315263748 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08761779218912125 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30471551418304443 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3166225850582123 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_7‚0.5 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01769890822470188 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9935848712921143 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973679184913635 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.28125 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 (particle/f_score‚0.33900412917137146 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2648148536682129 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3263888955116272 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3266666531562805 particle/accuracy_5_of_7‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010449734516441822 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3823140561580658 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908666610717773 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3184475600719452 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11936938017606735 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9902437329292297 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3581526577472687 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4107232391834259 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3060060143470764 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32716047763824463 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.34992218017578125 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08855465054512024 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09838346391916275 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931235909461975 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07295037060976028 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34490081667900085 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00432605342939496 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0840267688035965 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984840452671051 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17567098140716553 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.34666669368743896 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.37188470363616943 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08529410511255264 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9957366585731506 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚2417 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.28773948550224304 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.24621213972568512 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08595388382673264 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.0833333358168602 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970719814300537 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03005390800535679 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 7validation_average_jet_accuracy‚0.37650156021118164 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2708333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918187856674194 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10402939468622208 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2965277433395386 particle/accuracy_5_of_5‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 particle/accuracy_4_of_6‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 _runtime‚2050.848766633 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.29411765933036804 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.019284334033727646 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6353591680526733 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4087912440299988 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3470211625099182 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1329667568206787 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9861886501312256 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921278357505798 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0792740061879158 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1706896424293518 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3228478729724884 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.36851707100868225 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03213513642549515 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3294573724269867 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32815831899642944 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35956788063049316 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34861889481544495 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15534010529518127 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.10101010650396347 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2944372296333313 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.386427640914917 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_6_of_6‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 )loss/val_total_loss‚49.59603911378297 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32545554637908936 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.20714354515075684 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 epoch‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06031549721956253 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1336820274591446 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3581896722316742 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.205276221036911 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13912618160247803 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3639942705631256 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3567653298377991 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9924346208572388 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3583877682685852 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3420194089412689 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011855517514050007 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14353124797344208 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09795603156089783 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4288192689418793 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9495377540588379 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2828831374645233 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9936216473579407 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.34072479605674744 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2793154716491699 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07471264898777008 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4385964870452881 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9704108834266663 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9850582480430603 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12614941596984863 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.28333333134651184 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4123777151107788 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24503639340400696 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4132075309753418 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17356806993484497 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09123938530683517 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08909586817026138 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15046295523643494 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3422619104385376 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.34259259700775146 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8906042575836182 *jet/accuracy_2_of_5‚0.4007551670074463 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 particle/accuracy_2_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3713856041431427 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4260083734989166 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011342248879373074 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4145820438861847 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03915250673890114 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2962605357170105 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1458931267261505 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3125000298023224 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4426020383834839 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3721264600753784 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3711968660354614 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9889510273933411 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3379109501838684 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00480525242164731 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650321006775 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013555051758885384 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14409135282039642 particle/accuracy_1_of_1‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3125000298023224 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3121362626552582 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2622710168361664 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2786833345890045 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25899091362953186 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5025666952133179 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3764769732952118 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25445207953453064 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 _step‚197 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.39547115564346313 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_3‚0.0167212076485157 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_1_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857977628707886 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4294933080673218 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4710855782032013 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0917193666100502 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3461494743824005 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.41787660121917725 +jet/accuracy_2_of_2‚0.09995010495185852 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21298165619373322 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,jet/accuracy_4_of_4‚0.005002059042453766 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 jet/accuracy_7_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2015378624200821 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.28947368264198303 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_2_of_6‚0.4166666865348816 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.38898810744285583 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07818930596113205 particle/accuracy_1_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04772064834833145 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11140350997447968 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.29385969042778015 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.34522339701652527 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3740774393081665 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08994010835886002 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11730600893497467 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 jet/accuracy_4_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3899993896484375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12939448654651642 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3345890939235687 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13694152235984802 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14828889071941376 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03035714291036129 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.0533333346247673 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.09615384787321091 % _timestamp‚1.7884396108020687e+09 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39089149236679077 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9958959817886353 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3684210479259491 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_5‚0.09650538116693497 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49794742465019226 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0347222238779068 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9640803933143616 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.39547115564346313 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1F[\ÙÊBt/event_purity‚0.4004172384738922 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6455504894256592 particle/accuracy_2_of_4‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8236193656921387 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0434676818549633 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.32623952627182007 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 particle/accuracy_4_of_7‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09919361025094986 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.212301567196846 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3800044357776642 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0881667509675026 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.40572962164878845 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3622436225414276 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06310398876667023 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.29810404777526855 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10712031275033951 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.38957032561302185 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0804818719625473 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6818963289260864 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4049404263496399 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15667970478534698 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8206283450126648 jet/accuracy_3_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08761779218912125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887885451316833 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09238982945680618 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05603448301553726 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4114583432674408 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3060606122016907 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.319324791431427 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ­Ì(lÖØ¥ & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.37188470363616943 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08529410511255264 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9957366585731506 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚2417 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.28773948550224304 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.24621213972568512 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08595388382673264 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.0833333358168602 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970719814300537 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03005390800535679 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 7validation_average_jet_accuracy‚0.37650156021118164 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2708333432674408 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918187856674194 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10402939468622208 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2965277433395386 particle/accuracy_5_of_5‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 particle/accuracy_4_of_6‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 _runtime‚2050.848766633 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.29411765933036804 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.019284334033727646 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6353591680526733 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4087912440299988 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3470211625099182 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1329667568206787 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9861886501312256 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921278357505798 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0792740061879158 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1706896424293518 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3228478729724884 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.36851707100868225 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03213513642549515 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3294573724269867 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32815831899642944 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35956788063049316 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34861889481544495 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15534010529518127 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.10101010650396347 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2944372296333313 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.386427640914917 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_6_of_6‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 )loss/val_total_loss‚49.59603911378297 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32545554637908936 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.20714354515075684 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 epoch‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06031549721956253 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1336820274591446 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3581896722316742 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.205276221036911 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13912618160247803 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3639942705631256 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3567653298377991 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9924346208572388 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3583877682685852 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3420194089412689 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011855517514050007 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14353124797344208 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09795603156089783 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 particle/accuracy_3_of_7‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4288192689418793 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9495377540588379 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2828831374645233 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9936216473579407 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.34072479605674744 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2793154716491699 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07471264898777008 particle/accuracy_2_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4385964870452881 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9704108834266663 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9850582480430603 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12614941596984863 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.28333333134651184 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4123777151107788 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24503639340400696 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4132075309753418 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17356806993484497 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09123938530683517 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08909586817026138 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15046295523643494 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3422619104385376 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.34259259700775146 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8906042575836182 *jet/accuracy_2_of_5‚0.4007551670074463 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 particle/accuracy_2_of_7‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3713856041431427 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4260083734989166 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011342248879373074 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4145820438861847 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03915250673890114 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2962605357170105 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1458931267261505 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3125000298023224 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4426020383834839 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3721264600753784 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3711968660354614 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9889510273933411 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3379109501838684 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00480525242164731 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650321006775 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013555051758885384 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14409135282039642 particle/accuracy_1_of_1‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3125000298023224 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3121362626552582 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2622710168361664 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2786833345890045 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25899091362953186 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5025666952133179 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3764769732952118 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25445207953453064 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 _step‚197 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.39547115564346313 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_3‚0.0167212076485157 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_1_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857977628707886 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4294933080673218 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4710855782032013 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0917193666100502 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3461494743824005 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.41787660121917725 +jet/accuracy_2_of_2‚0.09995010495185852 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21298165619373322 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,jet/accuracy_4_of_4‚0.005002059042453766 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 jet/accuracy_7_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2015378624200821 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.28947368264198303 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_2_of_6‚0.4166666865348816 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.38898810744285583 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07818930596113205 particle/accuracy_1_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04772064834833145 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11140350997447968 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.29385969042778015 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.34522339701652527 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3740774393081665 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08994010835886002 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11730600893497467 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 jet/accuracy_4_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3899993896484375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12939448654651642 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3345890939235687 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13694152235984802 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14828889071941376 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03035714291036129 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.0533333346247673 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.09615384787321091 % _timestamp‚1.7884396108020687e+09 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39089149236679077 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9958959817886353 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3684210479259491 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_5‚0.09650538116693497 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49794742465019226 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0347222238779068 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9640803933143616 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.39547115564346313 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4004172384738922 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.6455504894256592 particle/accuracy_2_of_4‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8236193656921387 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0434676818549633 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.32623952627182007 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 particle/accuracy_4_of_7‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09919361025094986 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.212301567196846 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3800044357776642 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0881667509675026 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.40572962164878845 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3622436225414276 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06310398876667023 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.29810404777526855 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10712031275033951 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.38957032561302185 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0804818719625473 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6818963289260864 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4049404263496399 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15667970478534698 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8206283450126648 jet/accuracy_3_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08761779218912125 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRE¥ª\·&qqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887885451316833 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.36666667461395264 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09238982945680618 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05603448301553726 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4114583432674408 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3060606122016907 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.319324791431427 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3170037269592285 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13445228338241577 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18703702092170715 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.2599050998687744 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3258002996444702 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17110976576805115 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.04462890699505806 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9815108180046082 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04325397312641144 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_7_of_7‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19532573223114014 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29925599694252014 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.7639520168304443 &particle/accuracy‚0.84184330701828 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4089580178260803 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3476034700870514 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48301976919174194 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12278897315263748 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08761779218912125 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30471551418304443 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3166225850582123 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_7‚0.5 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01769890822470188 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9935848712921143 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973679184913635 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.28125 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 (particle/f_score‚0.33900412917137146 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2648148536682129 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3263888955116272 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3266666531562805 particle/accuracy_5_of_7‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010449734516441822 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3823140561580658 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908666610717773 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3184475600719452 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11936938017606735 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9902437329292297 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3581526577472687 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4107232391834259 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3060060143470764 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32716047763824463 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.34992218017578125 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08855465054512024 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09838346391916275 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931235909461975 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07295037060976028 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34490081667900085 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00432605342939496 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0840267688035965 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984840452671051 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17567098140716553 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.34666669368743896 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932ÅlY²Kׂ(:… ¾ØåÔ ÏþÈ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚32.0 gpu.0.powerWatts‚77.001' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.4002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ÒÁ¡&Ø‚(:› ¾ØåÔ§†„Ì disk.rbd2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 4622.65625 disk.sda2.out‚0! disk.nvme0n1.in‚ 1724.57421875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-0.in‚ 1724.57421875" disk.dm-0.out‚17682.0869140625) disk./.usagePercent‚74.26201271321126! disk.rbd2.out‚116.07666015625 disk.sda.out‚ 86.69140625 disk.sda2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 4.941539e+06 network.recv‚781935% disk./.usageGB‚1106.5576553344727 disk.sda.in‚0 disk.dm-1.out‚ 86.69140625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda1.out‚0% disk.nvme0n1.out‚17682.0869140625$ memory_percent‚99.97251033782959 disk.rbd3.in‚0 disk.sda3.out‚ 86.69140625 disk.rbd0.in‚0 disk.drbd1777.in‚0& proc.memory.availableMB‚ 4.50390625* proc.memory.percent‚28.214454650878906 cpu‚25.329209832900354 disk.sda1.in‚0' disk.nvme0c0n1.out‚17682.0869140625 disk.drbd1777.out‚0yDöؓق(:ˆ ÍØåÔþ“žÉ gpu.0.gpu‚81.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚42.0 gpu.0.powerWatts‚120.682' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1364 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’¢-ü Ú‚(:• ÍØåÔÑ¡àÊ proc.cpu.threads‚17 network.sent‚ 5.065254e+06 network.recv‚788558 disk.rbd3.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2753.60546875 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 87.453125# disk.nvme0c0n1.in‚ 1724.57421875 disk.drbd1777.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚77.13453769683838 disk.rbd1.out‚ 0.919921875! disk.rbd2.out‚116.58837890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.57421875) proc.memory.availableMB‚ 3746.27734375% disk./.usageGB‚1106.3003959655762 disk.rbd1.in‚0% disk.nvme0c0n1.out‚23184.12109375 disk.dm-1.out‚ 87.453125 disk.rbd2.in‚0) disk./.usagePercent‚74.24474782110964 disk.sda.in‚0 disk.dm-0.out‚23184.12109375 disk.sda.out‚ 87.453125 disk.sda2.out‚0! disk.nvme0n1.in‚ 1724.57421875# disk.nvme0n1.out‚23184.12109375* proc.memory.percent‚16.806674003601074 cpu‚25.326886981268384ûœóDžÛ‚(:“ ÜØåÔõ¹É gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚259.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.821799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â½.Ü‚(:£ ÜØåÔ¢¢ÕË network.sent‚ 5.071024e+06) disk./.usagePercent‚74.24474628506444% disk./.usageGB‚1106.3003730773926 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚23283.49072265625* proc.memory.percent‚16.806674003601074 disk.rbd1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1724.57421875( disk.nvme0c0n1.out‚23283.49072265625 network.recv‚793235 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚23283.49072265625 disk.dm-1.out‚ 88.06640625 disk.sda.out‚ 88.06640625 disk.sda3.out‚ 88.06640625 disk.dm-0.in‚ 1724.57421875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0# memory_percent‚77.1345853805542 disk.drbd1777.out‚0# disk.nvme0c0n1.in‚ 1724.57421875) proc.memory.availableMB‚ 3746.26953125 cpu‚25.324897453652813 proc.cpu.threads‚17! disk.rbd2.out‚116.91259765625 disk.rbd3.in‚0# proc.memory.rssMB‚ 2753.60546875âQv2Ý- _wandbruntime‚2086  _runtime‚2086a§8ÉÞà _step‚198 ' lr-AdamW/pg1‚0.00037522869662481184 ' lr-AdamW/pg2‚0.00037522869662481184 trainer/global_step‚2449 % _timestamp‚1.7884396460754206e+09 _runtime‚2086.120054441­ã{ªÎßÈ _step‚198 ' lr-AdamW/pg1‚0.00037522869662481184 ' lr-AdamW/pg2‚0.00037522869662481184 trainer/global_step‚2449 % _timestamp‚1.7884396460754206e+09 _runtime‚2086.120054441ƦNj 2à- _wandbruntime‚2086  _runtime‚2086iò3ú'áôO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4550763766272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4548163557376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273930289934765 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269389674597376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23531751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45764880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278332199720448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚273781409562624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36258395 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17031324 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240452251481088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884396460760841e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40562637 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234215457574912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23531754 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70337024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269389700025344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17030886 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45744847 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278329573120000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234215485709312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6236723569152 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚2086.120921945 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4550763766272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45764880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278329573120000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273781409562624 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚273781435954176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240452181144064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6236765771776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36258395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 trainer/global_step‚2449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36258401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543213622053 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273932913647397±ZœCÄâùO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4550763766272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45764880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278329573120000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273781409562624 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚273781435954176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240452181144064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6236765771776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36258395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 trainer/global_step‚2449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36258401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543213622053 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273932913647397 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4550763766272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4548163557376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273930289934765 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269389674597376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23531751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45764880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278332199720448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚273781409562624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36258395 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17031324 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240452251481088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884396460760841e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6Deš°"4; viceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40562637 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234215457574912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23531754 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70337024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269389700025344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17030886 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45744847 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278329573120000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234215485709312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6236723569152 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚2086.120921945ÇÒ°G2ã- _wandbruntime‚2086  _runtime‚2086¼nQ'äþM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4552596108288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274041084164545 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45783140 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6239322763776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274038460451705 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4542444680057 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82055777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278441971767808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240547196526592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36272637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269496041199616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273889375659520 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234307845628416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6239278477312 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40578514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278441971767808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚273889349267968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36272631 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278439345154048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23540950 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36272631 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 _runtime‚2086.868501261 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36272637 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4549995886080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17037567 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45763081 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273889375659520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4549995886080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234307873762816 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23540947 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096om† 'åƒN G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82055777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278441971767808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240547196526592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36272637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269496041199616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273889375659520 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234307845628416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6239278477312 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40578514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278441971767808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚273889349267968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36272631 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278439345154048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23540950 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36272631 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82035712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 _runtime‚2086.868501261 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36272637 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4549995886080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17037567 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45763081 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273889375659520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4549995886080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234307873762816 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23540947 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4552596108288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274041084164545 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45783140 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6239322763776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274038460451705 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4542444680057ÈN)¹˜2æ- _wandbruntime‚2086  _runtime‚2086¢2?QSçÍ 0loss/FBt2/assignment_loss‚0.4111499756574631 epoch‚2 1loss/FBt1/assignment_loss‚0.15765684429739918 %loss/total_loss‚48.87964604733642 % _timestamp‚1.7884396468254242e+09 2loss/SRqqt3/assignment_loss‚2.6554246153150287 2loss/SRqqt4/assignment_loss‚2.4444885065867785 /loss/FRt1/detection_loss‚1.7624727971241123 _step‚201 /loss/FRt1/assignment_loss‚2.447417682311574 0loss/FBt4/assignment_loss‚0.1530602391809225 .loss/FRt2/detection_loss‚2.141236489904778 /loss/FRt3/detection_loss‚2.5608076407359195 2loss/SRqqt2/assignment_loss‚2.0293280185531763 1loss/SRqqt1/detection_loss‚1.1385393468397005 /loss/SRqqt2/detection_loss‚1.77795629380113 _runtime‚2086.869998492 0loss/FBt3/assignment_loss‚0.6762971392917362 -loss/FRt4/detection_loss‚2.51165072247386 .loss/FBt1/detection_loss‚0.645718373910041 /loss/FBt2/detection_loss‚2.3147814989089968 .loss/FBt4/detection_loss‚3.350354778766632 0loss/FRt4/assignment_loss‚3.5172106698155403 2loss/SRqqt1/assignment_loss‚1.4389945827424526 1loss/SRqqt3/detection_loss‚2.1030211806297303 /loss/FRt2/assignment_loss‚3.928076222538948 /loss/FRt3/assignment_loss‚4.185034806911762 0loss/SRqqt4/detection_loss‚2.290341818261714 -loss/FBt3/detection_loss‚2.23862580277703 trainer/global_step‚2449¾ÆÅèXèÒ 0loss/FBt2/assignment_loss‚0.4111499756574631 epoch‚2 1loss/FBt1/assignment_loss‚0.15765684429739918 %loss/total_loss‚48.87964604733642 % _timestamp‚1.7884396468254242e+09 2loss/SRqqt3/assignment_loss‚2.6554246153150287 2loss/SRqqt4/assignment_loss‚2.4444885065867785 /loss/FRt1/detection_loss‚1.7624727971241123 _step‚201 /loss/FRt1/assignment_loss‚2.447417682311574 0loss/FBt4/assignment_loss‚0.1530602391809225 .loss/FRt2/detection_loss‚2.141236489904778 /loss/FRt3/detection_loss‚2.5608076407359195 2loss/SRqqt2/assignment_loss‚2.0293280185531763 1loss/SRqqt1/detection_loss‚1.1385393468397005 /loss/SRqqt2/detection_loss‚1.77795629380113 _runtime‚2086.869998492 0loss/FBt3/assignment_loss‚0.6762971392917362 -loss/FRt4/detection_loss‚2.51165072247386 .loss/FBt1/detection_loss‚0.645718373910041 /loss/FBt2/detection_loss‚2.3147814989089968 .loss/FBt4/detection_loss‚3.350354778766632 0loss/FRt4/assignment_loss‚3.5172106698155403 2loss/SRqqt1/assignment_loss‚1.4389945827424526 1loss/SRqqt3/detection_loss‚2.1030211806297303 /loss/FRt2/assignment_loss‚3.928076222538948 /loss/FRt3/assignment_loss‚4.185034806911762 0loss/SRqqt4/detection_loss‚2.290341818261714 -loss/FBt3/detection_loss‚2.23862580277703 trainer/global_step‚2449ɨTø“é‚(:ˆ ëØåÔŽ™‰É gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚257.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìNzé-ê‚(:¢ ëØåÔ㶖Ì& disk.nvme0n1.out‚23283.81103515625) proc.memory.availableMB‚ 3746.01171875 cpu‚25.3228288895767 network.recv‚797471 disk.rbd1.out‚ 0.919921875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 88.64453125% disk./.usageGB‚1106.3003730773926 disk.drbd1777.out‚0 disk.sda.out‚ 88.64453125* proc.memory.percent‚16.806674003601074 proc.cpu.threads‚17) disk./.usagePercent‚74.24474628506444 disk.drbd1777.in‚0! disk.nvme0n1.in‚ 1724.57421875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 88.64453125 disk.sda1.in‚0$ memory_percent‚77.13615894317627# proc.memory.rssMB‚ 2753.60546875# disk.dm-0.out‚23283.81103515625 network.sent‚ 5.077224e+06# disk.nvme0c0n1.in‚ 1724.57421875( disk.nvme0c0n1.out‚23283.81103515625! disk.rbd2.out‚116.91259765625 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1724.57421875 disk.sda1.out‚0ÍëB ë‚(:’ úØåÔðÑ É gpu.0.gpu‚82.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚239.252' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.85040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"¥9&"ì‚(:— úØåÔœøÊÌ disk.sda3.out‚ 89.06640625 network.sent‚ 5.185071e+06 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2753.60546875* proc.memory.percent‚16.806674003601074 cpu‚25.321024615771755 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚74.2447094199795% disk./.usageGB‚1106.2998237609863 network.recv‚802904% disk.nvme0c0n1.out‚23383.53515625 disk.dm-0.in‚ 1724.57421875 disk.dm-0.out‚23383.53515625 disk.dm-1.in‚0 disk.dm-1.out‚ 89.06640625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚116.99853515625 disk.sda.out‚ 89.06640625# disk.nvme0c0n1.in‚ 1724.57421875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1724.57421875 disk.rbd0.out‚0$ memory_percent‚77.13565826416016& proc.memory.availableMB‚ 3746.09375 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.919921875# disk.nvme0n1.out‚23383.53515625"ÁTh2í- _wandbruntime‚2125  _runtime‚2125Ú)Äî¾ % lr-AdamW/pg2‚0.000374229668863375 trainer/global_step‚2499 $ _timestamp‚1.788439685341023e+09 _runtime‚2125.385695195 _step‚202 % lr-AdamW/pg1‚0.000374229668863375ýUA¦Éïà _step‚202 % lr-AdamW/pg1‚0.000374229668863375 % lr-AdamW/pg2‚0.000374229668863375 trainer/global_step‚2499 $ _timestamp‚1.788439685341023e+09 _runtime‚2125.385695195ÊÝ駎2ð- _wandbruntime‚2125  _runtime‚2125SwÊ.ÉñóO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279178421221376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23991554 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36970201 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36970201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46679932 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17371938 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279178394829824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41363492 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6363914469376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pæj™Ã0ool.current‚42202624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 trainer/global_step‚2499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46659899 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279178421221376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46659899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41363062 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245198802853888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚203 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46679932 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245198732516864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283820815836672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4642394615296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283820815836672 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70337024 _runtime‚2125.386720778 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279338829372973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83630094 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238834860250112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283818189236224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283818189236224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4639794406400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279178394829824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4642394615296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36970195 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279178394829824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41363492 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6363914469376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 trainer/global_step‚2499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46659899 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279178421221376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46659899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41363062 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245198802853888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚203 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46679932 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245198732516864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283820815836672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4642394615296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283820815836672 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70337024 _runtime‚2125.386720778 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279338829372973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83630094 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238834860250112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283818189236224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283818189236224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4639794406400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279178394829824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4642394615296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36970195 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23991554 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36970201 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36970201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376ËÙ‚žò2ó- _wandbruntime‚2126  _runtime‚2126W1¬h'ôþM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83662564 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279286334535168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41378958 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238927248303616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 trainer/global_step‚2499 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4641626735104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279286360926720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24000747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41379369 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36984437 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245293725785600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283927961270272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46678133 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46698192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17378619 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4636523689537 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274813099913216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274813074485248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245293653364736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4644226957312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36984431 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 _step‚204 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6366405061120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 % _timestamp‚1.7884396860771394e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279446999889913 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36984431 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279449623602753 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2126.122268047 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4641626735104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17378211 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46678133 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46698192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283930587884032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279286360926720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283930587884032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36984437 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4644226957312÷Áv\yõƒN ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24000750 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4644226957312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83662564 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279286334535168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41378958 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238927248303616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 trainer/global_step‚2499 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4641626735104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279286360926720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24000747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41379369 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36984437 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245293725785600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283927961270272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46678133 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46698192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17378619 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4636523689537 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274813099913216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274813074485248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245293653364736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4644226957312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36984431 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 _step‚204 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6366405061120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 % _timestamp‚1.7884396860771394e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279446999889913 G:-²g7DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36984431 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279449623602753 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2126.122268047 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4641626735104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17378211 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46678133 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46698192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283930587884032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279286360926720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283930587884032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36984437 FLžý‚(:“ §ÙåÔ†ÈÉ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚162.921' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.584199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—`ò%þ‚(:š §ÙåÔ–÷³Ë disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚117.0927734375 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.57421875 disk.sda3.out‚ 90.703125# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 cpu‚25.315476177829137 disk.dm-0.in‚ 1724.57421875) disk./.usagePercent‚74.24471146803977% disk.nvme0n1.out‚23582.4833984375 disk.sda3.in‚0" disk.dm-0.out‚23582.4833984375# proc.memory.rssMB‚ 2753.60546875* proc.memory.percent‚16.806674003601074 proc.cpu.threads‚17 network.recv‚817973 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 90.703125) proc.memory.availableMB‚ 3745.53515625 disk.rbd1.out‚ 0.919921875$ memory_percent‚77.13906764984131% disk./.usageGB‚1106.2998542785645 disk.sda.out‚ 90.703125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚23582.4833984375 network.sent‚ 5.305386e+06‡„¿C2ÿ- _wandbruntime‚2165  _runtime‚2165‚ø/aȀ ' lr-AdamW/pg1‚0.00037319303087804607 ' lr-AdamW/pg2‚0.00037319303087804607 trainer/global_step‚2549 $ _timestamp‚1.788439724962837e+09 _runtime‚2165.007486106 _step‚206nòoÍÇ _runtime‚2165.007486106 _step‚206 ' lr-AdamW/pg1‚0.00037319303087804607 ' lr-AdamW/pg2‚0.00037319303087804607 trainer/global_step‚2549 $ _timestamp‚1.788439724962837e+09Î(…Ÿ}2‚- _wandbruntime‚2165  _runtime‚2165¹øú'ƒôO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289306805352448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249945206490112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17713956 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284575380097024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47574951 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280023792024576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42164852 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚207 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243454291059712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6490987864576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284575380097024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37681995 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284749992523813 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47594984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 _runtime‚2165.008452567 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47574951 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284575406488576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24451351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17713501 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4731425255424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284747368811181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280023817452544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289309431952896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289306805352448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37682001 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4734025464320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284575406488576 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37682001 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42165310 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2549 % _timestamp‚1.7884397249635658e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6490943564800 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4726175071269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4723576786605 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4731425255424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37681995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0{œ]ÿ'„ùO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17713956 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284575380097024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47574951 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280023792024576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42164852 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚207 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243454291059712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6490987864576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284575380097024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37681995 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284749992523813 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47594984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 _runtime‚2165.008452567 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47574951 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284575406488576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24451351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17713501 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4731425255424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284747368811181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280023817452544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289309431952896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289306805352448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85256946 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37682001 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4734025464320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284575406488576 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37682001 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42165310 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2549 % _timestamp‚1.7884397249635658e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6490943564800 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4726175071269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4723576786605 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4731425255424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37681995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289306805352448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249945206490112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÏÓDõg2…- _wandbruntime‚2165  _runtime‚2165; ¾¡²†ýM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42181365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24460547 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280130158626816 J=DeviceStatsMonitê™»äQ%or.on_train_batch_end/active.small_pool.freed‚47593185 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚2549 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42180937 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24460550 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284855539328121 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243546679113216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 _runtime‚2165.772274526 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284858163040961 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4735857806336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284683346193920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37696237 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17720815 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289419204000256 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4735857806336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289419204000256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37696237 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250040180179456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 $ _timestamp‚1.788439725727093e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ŠGÉ1'‡‚N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17720815 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289419204000256 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4735857806336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289419204000256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37696237 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250040180179456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 $ _timestamp‚1.788439725727093e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42181365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24460547 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280130158626816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47593185 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚2549 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42180937 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24460550 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284855539328121 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243546679113216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 _runtime‚2165.772274526 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284858163040961 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4735857806336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284683346193920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37696237 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647ÐQ°¹2ˆ- _wandbruntime‚2165  _runtime‚2165yþÍ[‰Õ 1loss/SRqqt4/detection_loss‚1.9730165880918502 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚1.2921702303495761 2loss/SRqqt2/assignment_loss‚1.8910721829324437 2loss/SRqqt3/assignment_loss‚2.1683267429471016 /loss/FBt3/detection_loss‚2.2891312420368193 0loss/FRt1/assignment_loss‚2.2187036752700804 0loss/FBt4/assignment_loss‚0.5527276886627078 .loss/FBt4/detection_loss‚2.176494985818863 2loss/SRqqt4/assignment_loss‚1.7719012263417244 .loss/FRt3/detection_loss‚2.347813657919566 /loss/FRt2/detection_loss‚2.3002961666687676 /loss/FBt1/detection_loss‚0.6797486445555433 %loss/total_loss‚46.97236897246347 _runtime‚2165.773731461 _step‚209 .loss/FRt2/assignment_loss‚3.38162982593412 1loss/FBt3/assignment_loss‚0.33394520338624717 1loss/SRqqt3/detection_loss‚2.6960781131471907 % _timestamp‚1.7884397257290332e+09 /loss/FRt3/assignment_loss‚4.332251107692718 -loss/FRt4/detection_loss‚2.30764628748099 /loss/FBt2/detection_loss‚3.2311946551005044 epoch‚2 /loss/FRt4/assignment_loss‚3.859912160038948 1loss/SRqqt2/detection_loss‚1.8884614873350711 1loss/FBt1/assignment_loss‚0.14072819868355216 2loss/FBt2/assignment_loss‚0.010959604755043983 /loss/FRt1/detection_loss‚2.1836473356187343 1loss/SRqqt1/detection_loss‚0.94451196169531032zˆ`ŠÚ /loss/FRt4/assignment_loss‚3.859912160038948 1loss/SRqqt2/detection_loss‚1.8884614873350711 1loss/FBt1/assignment_loss‚0.14072819868355216 2loss/FBt2/assignment_loss‚0.010959604755043983 /loss/FRt1/detection_loss‚2.1836473356187343 1loss/SRqqt1/detection_loss‚0.9445119616953103 1loss/SRqqt4/detection_loss‚1.9730165880918502 trainer/global_step‚2549 2loss/SRqqt1/assignment_loss‚1.2921702303495761 2loss/SRqqt2/assignment_loss‚1.8910721829324437 2loss/SRqqt3/assignment_loss‚2.1683267429471016 /loss/FBt3/detection_loss‚2.2891312420368193 0loss/FRt1/assignment_loss‚2.2187036752700804 0loss/FBt4/assignment_loss‚0.5527276886627078 .loss/FBt4/detection_loss‚2.176494985818863 2loss/SRqqt4/assignment_loss‚1.7719012263417244 .loss/FRt3/detection_loss‚2.347813657919566 /loss/FRt2/detection_loss‚2.3002961666687676 /loss/FBt1/detection_loss‚0.6797486445555433 %loss/total_loss‚46.97236897246347 _runtime‚2165.773731461 _step‚209 .loss/FRt2/assignment_loss‚3.38162982593412 1loss/FBt3/assignment_loss‚0.33394520338624717 1loss/SRqqt3/detection_loss‚2.6960781131471907 % _timestamp‚1.7884397257290332e+09 /loss/FRt3/assignment_loss‚4.332251107692718 -loss/FRt4/detection_loss‚2.30764628748099 /loss/FBt2/detection_loss‚3.2311946551005044 epoch‚2Ñ4ã ‹‚(:’ ¶ÙåÔÿ™‡É gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚232.45' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.489999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(á-&Œ‚(:› ¶ÙåÔ°¤øÊ disk.rbd3.in‚0 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2753.60546875 network.sent‚ 5.31062e+06 disk.sda3.in‚0' disk.nvme0c0n1.out‚23681.7861328125 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0* proc.memory.percent‚16.806674003601074 disk.sda1.in‚0) disk./.usagePercent‚74.24470250777607# disk.nvme0c0n1.in‚ 1724.57421875) proc.memory.availableMB‚ 1872.09765625 disk.sda2.in‚0! disk.nvme0n1.in‚ 1724.57421875 disk.sda3.out‚ 91.0078125 disk.dm-0.in‚ 1724.57421875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.out‚117.1552734375 cpu‚25.31367883052107 network.recv‚821683 disk.rbd3.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚23681.7861328125 disk.dm-1.out‚ 91.0078125 disk.rbd1.in‚0 disk.sda.out‚ 91.0078125% disk./.usageGB‚1106.2997207641602 disk.sda2.out‚0% disk.nvme0n1.out‚23681.7861328125 disk.drbd1777.in‚0 disk.sda1.out‚0$ memory_percent‚88.57362270355225£D•'‚(:’ ÅÙåÔû¶ØÈ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚43.0 gpu.0.powerWatts‚129.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.868000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®’BÉ(Ž‚(: ÅÙåÔÇîË disk.rbd1.out‚ 0.919921875 disk.sda.out‚ 91.7421875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 network.recv‚828198 disk.dm-1.out‚ 91.7421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚74.24470660389663" disk.dm-0.out‚23780.9189453125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1724.57421875% disk.nvme0n1.out‚23780.9189453125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 1724.57421875' disk.nvme0c0n1.out‚23780.9189453125 disk.rbd0.in‚0$ memory_percent‚88.57438564300537) proc.memory.availableMB‚ 1871.97265625# proc.memory.rssMB‚ 2753.60546875 disk.rbd2.out‚117.1591796875! disk.nvme0n1.in‚ 1724.57421875 cpu‚25.311893050128795 proc.cpu.threads‚17% disk./.usageGB‚1106.2997817993164 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 91.7421875 network.sent‚ 5.419275e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.806674003601074#ËêŠ2- _wandbruntime‚2204  _runtime‚2204¡o½†È _step‚210 ' lr-AdamW/pg1‚0.00037211899870262755 ' lr-AdamW/pg2‚0.00037211899870262755 trainer/global_step‚2599 $ _timestamp‚1.788439763990661e+09 _runtime‚2204.035359586kŒŒÉÍ‘Ç $ _timestamp‚1.788439763990661e+09 _runtime‚2204.035359586 _step‚210 ' lr-AdamW/pg1‚0.00037211899870262755 ' lr-AdamW/pg2‚0.00037211899870262755 trainer/global_step‚2599ÒÜãl2’- _wandbruntime‚2204  _runtime‚2204.#ĩדóO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42966577 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18055866 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294795421468672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48510036 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254691621621760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48490003 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285340876166144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285340850738176 _runtime‚2204.036330987 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289972391755776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290155908249389 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38393795 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289972391755776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4815057511213 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38393795 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289972365364224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38393801 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4823056104448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294795421468672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24911154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38393801 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48510036 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70337024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6617998223872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4823056104448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 $ _timestamp‚1.788439763991453e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289972365364224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADexo©Î" viceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294798048069120 trainer/global_step‚2599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4825656313344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24911151 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248073665600512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290158531962021 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294798048069120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48490003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254691691958784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248073693734912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6617956021248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4825656313344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289972391755776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290155908249389 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38393795 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289972391755776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4815057511213 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38393795 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289972365364224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38393801 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4823056104448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294795421468672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24911154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38393801 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48510036 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70337024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6617998223872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4823056104448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 $ _timestamp‚1.788439763991453e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289972365364224 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294798048069120 trainer/global_step‚2599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4825656313344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24911151 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248073665600512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290158531962021 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294798048069120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48490003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254691691958784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248073693734912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6617956021248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4825656313344 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42982636 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38408031 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290080305069568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2204.766886701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294907820116480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248166081788416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248166053654016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6620487551488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4827488655360 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4824888433152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290266702479169 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24920347 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 trainer/global_step‚2599 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254786541205504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38408037 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48508237 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4819485138753 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48528296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38408031 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18062691 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48508237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86916268 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6620529740800 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290080331461120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4824888433152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290080331461120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18062289 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290264078766329 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285447217340416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647W—¸…‡#—ƒN ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290264078766329 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285447217340416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290080305069568 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42982636 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38408031 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290080305069568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2204.766886701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294907820116480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248166081788416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248166053654016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6620487551488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4827488655360 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4824888433152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290266702479169 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24920347 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 trainer/global_step‚2599 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254786541205504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38408037 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48508237 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4819485138753 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48528296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38408031 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18062691 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48508237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86916268 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6620529740800 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290080331461120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4824888433152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290080331461120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18062289 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6Ô¥Ö?2˜- _wandbruntime‚2204  _runtime‚2204Íp Æ[™Õ /loss/FBt4/detection_loss‚2.6441448479890823 /loss/FRt1/detection_loss‚2.1846024562915165 1loss/SRqqt4/detection_loss‚1.8800183242222048 &loss/total_loss‚49.257946789007626 trainer/global_step‚2599 .loss/FRt2/assignment_loss‚3.51354054772124 2loss/SRqqt1/assignment_loss‚1.1759605674004112 1loss/FBt1/assignment_loss‚0.17947161170893722 /loss/FBt3/detection_loss‚2.4696156203746797 epoch‚2 _step‚213 2loss/SRqqt3/assignment_loss‚1.6473745729612268 1loss/SRqqt1/detection_loss‚1.1853227073119745 _runtime‚2204.768359767 0loss/FBt3/assignment_loss‚0.5502351707778871 /loss/FRt2/detection_loss‚2.3697344891879024 /loss/FRt3/assignment_loss‚4.458873696625233 1loss/SRqqt2/assignment_loss‚1.877913222116019 1loss/SRqqt4/assignment_loss‚2.152236614868326 % _timestamp‚1.7884397647237644e+09 0loss/FBt2/assignment_loss‚0.5839952729293145 1loss/SRqqt3/detection_loss‚3.0733045909715737 /loss/FBt1/detection_loss‚0.6381296811253191 /loss/FBt4/assignment_loss‚0.716382447630167 /loss/FRt4/detection_loss‚2.2652711216793504 /loss/FBt2/detection_loss‚3.6896496216456094 0loss/FRt1/assignment_loss‚2.2123760885662502 0loss/FRt4/assignment_loss‚3.6374889113182247 /loss/FRt3/detection_loss‚2.3193327598273754 1loss/SRqqt2/detection_loss‚1.8329718437577998ÚõkŽ`šÚ trainer/global_step‚2599 .loss/FRt2/assignment_loss‚3.51354054772124 2loss/SRqqt1/assignment_loss‚1.1759605674004112 1loss/FBt1/assignment_loss‚0.17947161170893722 /loss/FBt3/detection_loss‚2.4696156203746797 epoch‚2 _step‚213 2loss/SRqqt3/assignment_loss‚1.6473745729612268 1loss/SRqqt1/detection_loss‚1.1853227073119745 _runtime‚2204.768359767 0loss/FBt3/assignment_loss‚0.5502351707778871 /loss/FRt2/detection_loss‚2.3697344891879024 /loss/FRt3/assignment_loss‚4.458873696625233 1loss/SRqqt2/assignment_loss‚1.877913222116019 1loss/SRqqt4/assignment_loss‚2.152236614868326 % _timestamp‚1.7884397647237644e+09 0loss/FBt2/assignment_loss‚0.5839952729293145 1loss/SRqqt3/detection_loss‚3.0733045909715737 /loss/FBt1/detection_loss‚0.6381296811253191 /loss/FBt4/assignment_loss‚0.716382447630167 /loss/FRt4/detection_loss‚2.2652711216793504 /loss/FBt2/detection_loss‚3.6896496216456094 0loss/FRt1/assignment_loss‚2.2123760885662502 0loss/FRt4/assignment_loss‚3.6374889113182247 /loss/FRt3/detection_loss‚2.3193327598273754 1loss/SRqqt2/detection_loss‚1.8329718437577998 /loss/FBt4/detection_loss‚2.6441448479890823 /loss/FRt1/detection_loss‚2.1846024562915165 1loss/SRqqt4/detection_loss‚1.8800183242222048 &loss/total_loss‚49.257946789007626Õ™·ô“›‚(:ˆ ÔÙåÔÿ õÈ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.109' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û­¦\*œ‚(:Ÿ ÔÙåÔ¥«ðÊ* proc.memory.percent‚16.806674003601074) disk./.usagePercent‚74.24470660389663 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2753.60546875 proc.cpu.threads‚17' disk.nvme0c0n1.out‚23880.0205078125 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda3.out‚ 92.23828125$ memory_percent‚88.57433795928955) proc.memory.availableMB‚ 1871.98046875 network.sent‚ 5.425087e+06 disk.sda2.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚23880.0205078125 disk.drbd1777.out‚0 disk.sda.in‚0 cpu‚25.31013617382123% disk./.usageGB‚1106.2997817993164 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-0.in‚ 1724.57421875 network.recv‚832395 disk.sda.out‚ 92.23828125 disk.sda1.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.57421875 disk.rbd0.in‚0 disk.dm-1.out‚ 92.23828125 disk.rbd1.in‚0 disk.rbd2.out‚117.2373046875% disk.nvme0n1.out‚23880.0205078125 disk.sda3.in‚0áüûF‚(:’ ãÙåÔ»‘ÝÈ gpu.0.gpu‚57.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KGæk"ž‚(:— ãÙåÔžêóÊ disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 23880.6328125 disk.dm-1.out‚ 92.80859375 disk.rbd1.out‚ 0.919921875 disk.rbd2.out‚117.2373046875$ memory_percent‚88.57629299163818) disk./.usagePercent‚74.24472222035621 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.sda.out‚ 92.80859375" disk.nvme0n1.out‚ 23880.6328125% disk./.usageGB‚1106.3000144958496 disk.sda1.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.57421875 disk.sda3.in‚0 disk.rbd0.out‚0 cpu‚25.308398850153306 disk.sda3.out‚ 92.80859375 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.sent‚ 5.533962e+06 disk.sda.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2753.60546875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1724.57421875 disk.dm-0.out‚ 23880.6328125 disk.drbd1777.in‚0* proc.memory.percent‚16.806674003601074) proc.memory.availableMB‚ 1871.66015625 proc.cpu.threads‚17 network.recv‚839526 disk.sda1.in‚0~ÊÅŸ‚(:’ òÙåÔ•½É gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚240.532' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚48.10640000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§æÊ ‚(:” òÙåÔ¡¾Ë% disk.nvme0c0n1.out‚23979.75390625 disk.dm-1.out‚ 93.3515625 disk.drbd1777.out‚0 disk.rbd3.in‚0 cpu‚25.30668043251267 disk.sda2.in‚0 disk.sda3.out‚ 93.3515625 proc.cpu.threads‚17) disk./.usagePercent‚74.24472222035621 disk.rbd2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1106.3000144958496 disk.dm-1.in‚0 disk.sda1.out‚0$ memory_percent‚88.57488632202148* proc.memory.percent‚16.806674003601074 network.sent‚ 5.539251e+06! disk.nvme0n1.in‚ 1724.57421875# proc.memory.rssMB‚ 2753.60546875 disk.dm-0.in‚ 1724.57421875 disk.rbd1.out‚ 0.919921875 disk.rbd2.out‚118.3740234375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚23979.75390625 disk.sda3.in‚0' proc.memory.availableMB‚ 1871.890625 disk.rbd0.out‚0 network.recv‚843153 disk.dm-0.out‚23979.75390625 disk.drbd1777.in‚0 disk.sda.in‚0 disk.sda.out‚ 93.3515625# disk.nvme0c0n1.in‚ 1724.57421875á„]2¡- _wandbruntime‚2243  _runtime‚2243D¨Ç¢Á & lr-AdamW/pg1‚0.0003710077961638149 & lr-AdamW/pg2‚0.0003710077961638149 trainer/global_step‚2649 % _timestamp‚1.7884398031059492e+09 _runtime‚2243.150621307 _step‚214ÿÔõÌ£Æ trainer/global_step‚2649 % _timestamp‚1.7884398031059492e+09 _runtime‚2243.150621307 _step‚214 & lr-AdamW/pg1‚0.0003710077961638149 & lr-AdamW/pg2‚0.0003710077961638149Ö}/‚2¤- _wandbruntime‚2243  _runtime‚2243µ Ƭù'¥óO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39105595 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49425088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295369377022976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295567071400229 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290657934879744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300284037584896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 _runtime‚2243.151496096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6744642104832 trainer/global_step‚2649 _step‚215 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18397397 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753§eþŸþ'¦øO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚295369350631424 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25370951 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788439803106664e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300286664185344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39105595 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252693096410112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252693068275712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6744686404608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300284037584896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49405055 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259437710380544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290657909451776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4909136520485 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39105601 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4906538235821 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259437782814720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 _runtime‚2243.151496096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6744642104832 trainer/global_step‚2649 _step‚215 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18397397 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39105595 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49425088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295369377022976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295567071400229 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290657934879744 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300284037584896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43783801 % _timestamp‚1.7884398038421721e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300393809618944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4919119504384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25380150 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295477290336768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18403654 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295477316728320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocation8öjrIs.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4908367578489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25380147 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39119837 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚405 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290764276054016 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39119831 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49423289 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39119831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252785484463616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290764250626048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4910965863361 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18404059 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43784209 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300396436232704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4919119504384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6747188951040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300396436232704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 _runtime‚2243.887323441 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88543120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4916519282176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49443348 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295675241917377 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376yú; '©ƒN YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295675241917377 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43783801 % _timestamp‚1.7884398038421721e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300393809618944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88543120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4919119504384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25380150 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295477290336768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18403654 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295477316728320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4908367578489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25380147 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39119837 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚405 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290764276054016 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39119831 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49423289 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39119831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252785484463616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290764250626048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4910965863361 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18404059 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43784209 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300396436232704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4919119504384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6747188951040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300396436232704 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 _runtime‚2243.887323441 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88543120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4916519282176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49443348 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3Ø‹ú¶2ª- _wandbruntime‚2243  _runtime‚2243kæ>É\«Ö 0loss/SRqqt3/detection_loss‚2.536772100194808 1loss/SRqqt4/detection_loss‚1.9484806252850426 1loss/SRqqt1/detection_loss‚1.1570496351923794 /loss/FBt4/detection_loss‚2.0929701745510103 trainer/global_step‚2649 0loss/FRt4/assignment_loss‚3.3814058040463646 2loss/SRqqt4/assignment_loss‚1.7525029747850365 &loss/total_loss‚46.922527086571264 _step‚217 % _timestamp‚1.7884398038439088e+09 .loss/FRt1/detection_loss‚1.648124974258244 0loss/FBt2/assignment_loss‚0.6018369336088654 1loss/FBt3/assignment_loss‚0.18553066831082105 /loss/FBt1/detection_loss‚0.7012843134708545 /loss/FBt2/detection_loss‚3.0792305320501328 .loss/FBt3/detection_loss‚2.866750121116638 _runtime‚2243.888477445 0loss/FRt3/assignment_loss‚4.4971458406159375 1loss/FBt1/assignment_loss‚0.10290781597852383 /loss/FRt3/detection_loss‚2.2533567890976416 epoch‚2 /loss/FRt2/detection_loss‚2.5512491521683147 0loss/FRt1/assignment_loss‚1.4618367266654968 1loss/SRqqt2/assignment_loss‚2.033716281029311 1loss/SRqqt3/assignment_loss‚1.941391035914421 2loss/SRqqt1/assignment_loss‚1.1842118400672916 1loss/FBt4/assignment_loss‚0.31018840298056605 /loss/FRt2/assignment_loss‚4.083691061811244 /loss/FRt4/detection_loss‚2.3507498059161875 0loss/SRqqt2/detection_loss‚2.200143477456136Ç6°a¬Û /loss/FRt2/assignment_loss‚4.083691061811244 /loss/FRt4/detection_loss‚2.3507498059161875 0loss/SRqqt2/detection_loss‚2.200143477456136 0loss/SRqqt3/detection_loss‚2.536772100194808 1loss/SRqqt4/detection_loss‚1.9484806252850426 1loss/SRqqt1/detection_loss‚1.1570496351923794 /loss/FBt4/detection_loss‚2.0929701745510103 trainer/global_step‚2649 0loss/FRt4/assignment_loss‚3.3814058040463646 2loss/SRqqt4/assignment_loss‚1.7525029747850365 &loss/total_loss‚46.922527086571264 _step‚217 % _timestamp‚1.7884398038439088e+09 .loss/FRt1/detection_loss‚1.648124974258244 0loss/FBt2/assignment_loss‚0.6018369336088654 1loss/FBt3/assignment_loss‚0.18553066831082105 /loss/FBt1/detection_loss‚0.7012843134708545 /loss/FBt2/detection_loss‚3.0792305320501328 .loss/FBt3/detection_loss‚2.866750121116638 _runtime‚2243.888477445 0loss/FRt3/assignment_loss‚4.4971458406159375 1loss/FBt1/assignment_loss‚0.10290781597852383 /loss/FRt3/detection_loss‚2.2533567890976416 epoch‚2 /loss/FRt2/detection_loss‚2.5512491521683147 0loss/FRt1/assignment_loss‚1.4618367266654968 1loss/SRqqt2/assignment_loss‚2.033716281029311 1loss/SRqqt3/assignment_loss‚1.941391035914421 2loss/SRqqt1/assignment_loss‚1.1842118400672916 1loss/FBt4/assignment_loss‚0.31018840298056605Ù¯¹‡‘­‚(:† ÚåÔ±¶›É gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚249.44' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡@âÕ"®‚(:— ÚåÔ…‘ÅË disk.rbd1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚24078.91015625 network.sent‚ 5.545163e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-0.out‚24078.91015625 disk.dm-1.in‚0 disk.rbd2.out‚119.4794921875 disk.sda1.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2753.60546875 disk.sda.out‚ 93.8359375! disk.nvme0n1.in‚ 1724.57421875* proc.memory.percent‚16.806674003601074% disk.nvme0c0n1.out‚24078.91015625 cpu‚25.304869824161056 proc.cpu.threads‚17 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0 network.recv‚847438% disk./.usageGB‚1106.3000144958496 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1724.57421875 disk.dm-1.out‚ 93.8359375) disk./.usagePercent‚74.24472222035621 disk.drbd1777.in‚0 disk.sda3.out‚ 93.8359375 disk.rbd0.in‚0$ memory_percent‚88.57362270355225) proc.memory.availableMB‚ 1872.09765625äRž¯‚(:“ ÚåÔãØ–É gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚103.982' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.796400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ñ7š.°‚(:£ ÚåÔá¤ËË disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2753.60546875 cpu‚25.30320048424626 disk.sda.out‚ 94.52734375 disk.sda1.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚24178.06982421875# disk.dm-0.out‚24178.06982421875 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1724.57421875 disk.sda3.out‚ 94.52734375( disk.nvme0c0n1.out‚24178.06982421875 disk.dm-1.out‚ 94.52734375 network.recv‚853787 disk.dm-1.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1871.85546875) disk./.usagePercent‚74.24472222035621 disk.sda3.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0* proc.memory.percent‚16.806674003601074 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1724.57421875 disk.dm-0.in‚ 1724.57421875 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 0.919921875$ memory_percent‚88.57510089874268 network.sent‚ 5.653513e+06% disk./.usageGB‚1106.3000144958496 disk.sda1.in‚0! disk.rbd2.out‚119.63232421875‹+YA“±‚(:ˆ ŸÚåÔëÁ¥É gpu.0.gpu‚57.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—«Go²‚(:‹ ŸÚåÔ“¡ÄË disk.drbd1777.out‚0! disk.nvme0n1.in‚ 1724.57421875 network.recv‚858347 disk.sda3.in‚0 disk.sda3.out‚ 95.28515625 disk.rbd2.in‚0# proc.memory.rssMB‚ 2753.60546875 disk.dm-0.in‚ 1724.57421875 disk.dm-1.out‚ 95.28515625! disk.rbd2.out‚120.67529296875 disk.nvme0n1.out‚ 24178.875 network.sent‚ 5.659714e+06% disk./.usageGB‚1106.2991256713867 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 cpu‚25.301555347610115) disk./.usagePercent‚74.24466257060074 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.806674003601074 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 95.28515625# disk.nvme0c0n1.in‚ 1724.57421875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.out‚ 24178.875 disk.rbd1.out‚ 0.919921875 disk.dm-0.out‚ 24178.875# memory_percent‚88.5765790939331) proc.memory.availableMB‚ 1871.61328125êÉ$‰2³- _wandbruntime‚2282  _runtime‚2282V?ADÉ´Ã _step‚218 ' lr-AdamW/pg1‚0.00036985965483455176 ' lr-AdamW/pg2‚0.00036985965483455176 trainer/global_step‚2699 % _timestamp‚1.7884398419918125e+09 _runtime‚2282.036486061¥#÷ñÎµÈ trainer/global_step‚2699 % _timestamp‚1.7884398419918125e+09 _runtime‚2282.036486061 _step‚218 ' lr-AdamW/pg1‚0.00036985965483455176 ' lr-AdamW/pg2‚0.00036985965483455176ÚvPV2¶- _wandbruntime‚2282  _runtime‚2282²ÈÌË,!·óO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264184071067648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39817401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90157541 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295974993593344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5000617245093 trainer/global_step‚2699 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50340140 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5006317802496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305775280301568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264184143501824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90137502 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305772653701120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39817401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44567982 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300972987125805 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50320107 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300766362290176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295974968165376 _runtime‚2282.037453053 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300766335898624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18737228 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257312499085312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 $ _timestamp‚1.788439841992567e+09 _step‚219 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚305775280301568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257312470950912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39817395 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44567540 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50320107 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39817395 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5006317802496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300766335898624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18736789 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50340140 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25830754 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871644416512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 RCDeviceStatsMonitor.on_train_batch_start/rjt›Íequested_bytes.all.current‚ 2623712632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5008918011392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074§¶Ûþ'¸øO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39817395 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44567540 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50320107 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39817395 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5006317802496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300766335898624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18736789 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50340140 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25830754 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871644416512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5008918011392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264184071067648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39817401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90157541 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295974993593344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5000617245093 trainer/global_step‚2699 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50340140 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5006317802496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305775280301568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264184143501824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90137502 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305772653701120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39817401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44567982 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300972987125805 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50320107 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300766362290176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295974968165376 _runtime‚2282.037453053 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300766335898624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18737228 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257312499085312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 $ _timestamp‚1.788439841992567e+09 _step‚219 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚305775280301568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257312470950912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320Û“üí2¹- _wandbruntime‚2282  _runtime‚2282TÒ·Ú'ºþM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300874301995520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39831637 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44583440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25839950 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301081157642745 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300874301995520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300874275603968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305882425735168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25839947 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 _runtime‚2282.772960967 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39831637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305885052348928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257404887138816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296081334767616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301083781355585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50358400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50338341 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264279071648768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39831631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5010750353408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6874184509952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305885052348928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5010750353408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50358400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚419 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5002446587969 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5008150131200 % _timestamp‚1.7884398427276318e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18743493 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5008150131200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300874275603968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚220 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44583859 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18743909 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50338341 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257404859004416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296081309339648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0eUÆb '»ƒN XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4999848303097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚220 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44583859 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18743909 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50338341 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257404859004416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296081309339648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300874301995520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39831637 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44583440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25839950 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301081157642745 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300874301995520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300874275603968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305882425735168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25839947 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 _runtime‚2282.772960967 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39831637 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305885052348928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257404887138816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296081334767616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301083781355585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50358400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50338341 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264279071648768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39831631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5010750353408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6874184509952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305885052348928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5010750353408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50358400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚419 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5002446587969 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5008150131200 % _timestamp‚1.7884398427276318e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18743493 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5008150131200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300874275603968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51255192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _step‚223 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40529201 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301292052306944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306163321165824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301292026878976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51235159 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306384150276645 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45367665 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5097948651520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5089499685037 % _timestamp‚1.7884398811374564e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51255192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268930693682176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40529195 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311261269817344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2749 _runtime‚2321.182352145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40529195 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19077562 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311263896417792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26290554 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261931873626112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40529201 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306381526564013 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5100548860416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26290551 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6998745524736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19077114 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311261269817344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306163347557376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5100548860416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3ÁÜÿ'ÈùO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311261269817344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306163347557376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5100548860416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6998791921664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51255192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _step‚223 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40529201 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301292052306944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306163321165824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301292026878976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51235159 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306384150276645 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45367665 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5097948651520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5089499685037 % _timestamp‚1.7884398811374564e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51255192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268930693682176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40529195 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311261269817344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2749 _runtime‚2321.182352145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40529195 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19077562 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311263896417792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26290554 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261931873626112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40529201 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306381526564013 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5100548860416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26290551 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6998745524736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19077114 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ßdL‹2É- _wandbruntime‚2321  _runtime‚2321vàcìÊþM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40543437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚306271287262720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19084302 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301398368053248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40543431 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51253393 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262024261679616 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2749 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚311371041851392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5091329027705 _runtime‚2321.930058839 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45383637 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40543431 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5102381202432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26299750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B*¦²# 8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26299747 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚311373668465152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40543437 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5102381202432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269025607792640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269025533274624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2749 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚311371041851392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5091329027705 _runtime‚2321.930058839 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45383637 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40543431 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5102381202432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26299750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26299747 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚311373668465152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40543437 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5102381202432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269025607792640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269025533274624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40543437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚306271287262720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19084302 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301398368053248àøŽçT2Ì- _wandbruntime‚2321  _runtime‚2321&|ò4^ÍØ 2loss/SRqqt4/assignment_loss‚1.5729228776632522 1loss/SRqqt4/detection_loss‚2.1737429846306235 _runtime‚ 2321.9315347 _step‚225 /loss/FRt3/assignment_loss‚4.254351963315691 2loss/SRqqt1/assignment_loss‚1.2630138127067665 1loss/FBt2/assignment_loss‚0.08315648674033582 /loss/FBt4/detection_loss‚3.2784859027181352 0loss/FRt1/assignment_loss‚1.5920012058882878 /loss/FRt2/detection_loss‚2.8204838100232577 1loss/SRqqt1/detection_loss‚0.9679891259317667 trainer/global_step‚2749 /loss/FRt4/assignment_loss‚3.432184286415577 /loss/FBt1/detection_loss‚0.6402738324363569 0loss/FBt3/assignment_loss‚0.1642420694309029 /loss/FRt3/detection_loss‚2.7234338198389327 1loss/SRqqt3/detection_loss‚2.7381264659074636 /loss/FBt3/detection_loss‚3.4409123063087463 0loss/FRt2/assignment_loss‚3.3685723982359232 1loss/SRqqt2/detection_loss‚2.3683362596708797 2loss/SRqqt2/assignment_loss‚1.6809828482244327 2loss/SRqqt3/assignment_loss‚2.2492636167086086 1loss/FBt1/assignment_loss‚0.16812488929141078 /loss/FBt2/detection_loss‚3.1457405984401703 %loss/total_loss‚48.50528511952824 epoch‚2 1loss/FBt4/assignment_loss‚0.14202468656003475 /loss/FRt1/detection_loss‚1.6519271983154888 /loss/FRt4/detection_loss‚2.5849916741251944 % _timestamp‚1.7884398818866644e+09@ò@5cÎÝ /loss/FBt2/detection_loss‚3.1457405984401703 %loss/total_loss‚48.50528511952824 epoch‚2 1loss/FBt4/assignment_loss‚0.14202468656003475 /loss/FRt1/detection_loss‚1.6519271983154888 /loss/FRt4/detection_loss‚2.5849916741251944 % _timestamp‚1.7884398818866644e+09 2loss/SRqqt4/assignment_loss‚1.5729228776632522 1loss/SRqqt4/detection_loss‚2.1737429846306235 _runtime‚ 2321.9315347 _step‚225 /loss/FRt3/assignment_loss‚4.254351963315691 2loss/SRqqt1/assignment_loss‚1.2630138127067665 1loss/FBt2/assignment_loss‚0.08315648674033582 /loss/FBt4/detection_loss‚3.2784859027181352 0loss/FRt1/assignment_loss‚1.5920012058882878 /loss/FRt2/detection_loss‚2.8204838100232577 1loss/SRqqt1/detection_loss‚0.9679891259317667 trainer/global_step‚2749 /loss/FRt4/assignment_loss‚3.432184286415577 /loss/FBt1/detection_loss‚0.6402738324363569 0loss/FBt3/assignment_loss‚0.1642420694309029 /loss/FRt3/detection_loss‚2.7234338198389327 1loss/SRqqt3/detection_loss‚2.7381264659074636 /loss/FBt3/detection_loss‚3.4409123063087463 0loss/FRt2/assignment_loss‚3.3685723982359232 1loss/SRqqt2/detection_loss‚2.3683362596708797 2loss/SRqqt2/assignment_loss‚1.6809828482244327 2loss/SRqqt3/assignment_loss‚2.2492636167086086 1loss/FBt1/assignment_loss‚0.16812488929141078ágP’Ï‚(:‡ ÌÚåÔ®€®É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚263.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.713 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çt$Ђ(:™ ÌÚåÔçˆáË disk.sda.in‚0 disk.sda.out‚ 96.70703125 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 1724.578125 network.sent‚ 5.779549e+06 disk.sda3.out‚ 96.70703125# disk.dm-0.out‚24476.27392578125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚24476.27392578125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.068359375 disk.rbd2.out‚ 123.01953125) proc.memory.availableMB‚ 1871.76953125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1724.578125 disk.drbd1777.out‚0 disk.rbd2.in‚0 cpu‚25.29673719361619 disk.dm-1.out‚ 96.70703125# proc.memory.rssMB‚ 2753.60546875 disk.nvme0n1.in‚ 1724.578125 disk.dm-1.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.806674003601074 network.recv‚873432% disk./.usageGB‚1106.2991256713867& disk.nvme0n1.out‚24476.27392578125# memory_percent‚88.5756254196167) disk./.usagePercent‚74.24466257060074jÁ꟒Ñ‚(:‡ ÛÚåÔºé¸É gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚96.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o=ÒÛÒ‚(: ÛÚåÔ—¯£Ë disk.rbd3.in‚0# disk.dm-0.out‚24607.72998046875 proc.cpu.threads‚17% disk./.usageGB‚1106.3303489685059 disk.sda3.out‚97.46875( disk.nvme0c0n1.out‚24607.72998046875 disk.rbd1.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1871.73046875 network.recv‚879999 disk.nvme0n1.in‚ 1724.578125 disk.rbd0.in‚0 disk.rbd1.out‚ 1.068359375# memory_percent‚88.5758638381958 cpu‚25.295078635659944 disk.dm-1.in‚0 disk.dm-1.out‚97.46875 disk.drbd1777.out‚0 disk.rbd2.out‚124.0625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 1724.578125 disk.sda.out‚97.46875# proc.memory.rssMB‚ 2753.60546875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.578125 disk.sda1.in‚0* proc.memory.percent‚16.806674003601074) disk./.usagePercent‚74.24675799226848& disk.nvme0n1.out‚24607.72998046875 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 5.888477e+06I’Ð}žÓ‚(:“ êÚåÔ„êúÈ gpu.0.gpu‚60.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚165.889' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.177800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{êÔ‚(: êÚåÔòÔÝÊ disk.drbd1777.out‚0 disk.rbd2.out‚ 125.21875 disk.sda2.in‚0 network.sent‚ 5.8937e+06! disk.nvme0c0n1.in‚ 1724.578125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.1005859375 disk.sda2.out‚0 cpu‚25.29354740405257 disk.sda1.out‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚24608.126953125 network.recv‚883895% disk./.usageGB‚1106.3303489685059 disk.rbd0.out‚0 disk.sda.out‚ 98.05078125 disk.nvme0n1.in‚ 1724.578125) proc.memory.availableMB‚ 1871.71484375 proc.cpu.threads‚17 disk.dm-0.in‚ 1724.578125 disk.rbd2.in‚0 disk.rbd3.in‚0$ memory_percent‚88.57595920562744# proc.memory.rssMB‚ 2753.60546875 disk.rbd0.in‚0! disk.dm-0.out‚24608.126953125& disk.nvme0c0n1.out‚24608.126953125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0) disk./.usagePercent‚74.24675799226848 disk.sda1.in‚0 disk.sda3.out‚ 98.05078125* proc.memory.percent‚16.806674003601074 disk.dm-1.out‚ 98.05078125—^AW2Õ- _wandbruntime‚2359  _runtime‚2359ëÿÉÖà ' lr-AdamW/pg1‚0.00036745352053652746 ' lr-AdamW/pg2‚0.00036745352053652746 trainer/global_step‚2799 % _timestamp‚1.7884399199532113e+09 _runtime‚2359.997889488 _step‚226®OÎ×È ' lr-AdamW/pg1‚0.00036745352053652746 ' lr-AdamW/pg2‚0.00036745352053652746 trainer/global_step‚2799 % _timestamp‚1.7884399199532113e+09 _runtime‚2359.997889488 _step‚226⪠)2Ø- _wandbruntime‚2359  _runtime‚2359Õ;ú'ÙôO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273676918957056 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306609085592576 % _timestamp‚1.7884399199539843e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52150211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41241001 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311560306433024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5183578694309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5189579500544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5189579500544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52170244 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316752512534016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316749885933568 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 _step‚227 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5192179709440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311560332824576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52150211 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266551304435712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41241001 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52170244 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5192179709440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46169840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316749885933568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311560332824576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41240995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306609111020544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5180980409645 _runtime‚2359.998899812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46170300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311790066002221 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚311792689714853 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26750354 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41240995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19419489 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7125686955520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46169840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316749885933568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311560332824576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41240995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306609111020544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5180980409645 _runtime‚2359.998899812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46170300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311790066002221 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚311792689714853 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26750354 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41240995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19419489 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7125686955520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316862284581376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚228 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311668246138368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19426666 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316859657967616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41255231 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46186216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26759547 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚273771847991296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306715426766848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41255231 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273771920412160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52168445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5194012051456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266643664354816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5191411829248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚421 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311668272529920 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5182809752313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46185792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266643692489216 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5194012051456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93423676 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41255237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26759550 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52168445 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316862284581376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52188504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19426245 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306715452194816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5185408037185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311900860232001 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5191411829248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2799 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311668272529920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456b†c '݃N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316862284581376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚228 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311668246138368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19426666 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316859657967616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41255231 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46186216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26759547 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚273771847991296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306715426766848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41255231 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273771920412160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52168445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5194012051456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266643664354816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5191411829248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚421 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311668272529920 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5182809752313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46185792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266643692489216 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5194012051456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93423676 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41255237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26759550 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52168445 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316862284581376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52188504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19426245 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306715452194816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5185408037185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311900860232001 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5191411829248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2799 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311668272529920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93443741 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968äÀlm2Þ- _wandbruntime‚2360  _runtime‚23606Ë)Ž]ß× /loss/FRt2/assignment_loss‚3.566138411677161 1loss/FBt3/assignment_loss‚0.38921323825012555 0loss/SRqqt2/detection_loss‚1.919478355050087 epoch‚2 0loss/FRt4/assignment_loss‚3.4510477287013357 2loss/SRqqt1/assignment_loss‚1.3821013559970785 2loss/SRqqt2/assignment_loss‚1.8455159581452607 2loss/SRqqt4/assignment_loss‚1.9058625683711286 1loss/FBt1/assignment_loss‚0.19381301294370235 /loss/FRt3/detection_loss‚2.7752001881599426 0loss/FBt1/detection_loss‚0.43733457785009855 _step‚229 0loss/FRt1/assignment_loss‚1.9226805280361856 /loss/FRt3/assignment_loss‚4.430604543004717 % _timestamp‚1.7884399206923087e+09 2loss/SRqqt3/assignment_loss‚1.9515052424536812 1loss/SRqqt1/detection_loss‚1.0433132418376558 /loss/FBt4/detection_loss‚2.0672503958145776 %loss/total_loss‚46.47961555877116 _runtime‚2360.736890734 1loss/FBt2/assignment_loss‚0.24669700550536314 /loss/FRt2/detection_loss‚2.3835628108922826 1loss/SRqqt4/detection_loss‚1.8165059675250137 .loss/FBt2/detection_loss‚3.013781269391378 0loss/FBt4/assignment_loss‚0.6013541587938865 /loss/FRt1/detection_loss‚1.9383944505825639 /loss/FRt4/detection_loss‚2.6170899112050128 0loss/SRqqt3/detection_loss‚2.070437890291214 .loss/FBt3/detection_loss‚2.510732748291709 trainer/global_step‚2799}W—bàÜ 1loss/FBt2/assignment_loss‚0.24669700550536314 /loss/FRt2/detection_loss‚2.3835628108922826 1loss/SRqqt4/detection_loss‚1.8165059675250137 .loss/FBt2/detection_loss‚3.013781269391378 0loss/FBt4/assignment_loss‚0.6013541587938865 /loss/FRt1/detection_loss‚1.9383944505825639 /loss/FRt4/detection_loss‚2.6170899112050128 0loss/SRqqt3/detection_loss‚2.070437890291214 .loss/FBt3/detection_loss‚2.510732748291709 trainer/global_step‚2799 /loss/FRt2/assignment_loss‚3.566138411677161 1loss/FBt3/assignment_loss‚0.38921323825012555 0loss/SRqqt2/detection_loss‚1.919478355050087 epoch‚2 0loss/FRt4/assignment_loss‚3.4510477287013357 2loss/SRqqt1/assignment_loss‚1.3821013559970785 2loss/SRqqt2/assignment_loss‚1.8455159581452607 2loss/SRqqt4/assignment_loss‚1.9058625683711286 1loss/FBt1/assignment_loss‚0.19381301294370235 /loss/FRt3/detection_loss‚2.7752001881599426 0loss/FBt1/detection_loss‚0.43733457785009855 _step‚229 0loss/FRt1/assignment_loss‚1.9226805280361856 /loss/FRt3/assignment_loss‚4.430604543004717 % _timestamp‚1.7884399206923087e+09 2loss/SRqqt3/assignment_loss‚1.9515052424536812 1loss/SRqqt1/detection_loss‚1.0433132418376558 /loss/FBt4/detection_loss‚2.0672503958145776 %loss/total_loss‚46.47961555877116 _runtime‚2360.736890734å0W.cžá‚(:“ ùÚåÔ˜É gpu.0.gpu‚70.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚239.716' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.943200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÒL[â‚(:’ ùÚåÔ®øË network.recv‚887428! disk.dm-0.out‚24707.962890625 disk.rbd1.out‚ 1.1005859375 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚88.57579231262207 disk.sda2.in‚0 network.sent‚ 5.899445e+06% disk./.usageGB‚1106.3292503356934 disk.sda3.out‚ 98.59765625 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd2.out‚ 126.265625 disk.nvme0n1.in‚ 1724.578125$ disk.nvme0n1.out‚24707.962890625 disk.dm-0.in‚ 1724.578125# proc.memory.rssMB‚ 2753.60546875 cpu‚25.29201211991664) diœ$E†Âsk./.usagePercent‚74.24668426209863! disk.nvme0c0n1.in‚ 1724.578125& disk.nvme0c0n1.out‚24707.962890625 disk.dm-1.out‚ 98.59765625 disk.sda1.out‚0( proc.memory.availableMB‚ 1871.7421875* proc.memory.percent‚16.806674003601074 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 98.59765625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0¼Á{z“ã‚(:ˆ ˆÛåÔ™Û…É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚177.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àvqä‚(:‡ ˆÛåÔŽí›Ë disk.sda3.out‚99.0625 disk.rbd0.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚74.24671728707055 disk.rbd1.out‚ 1.16357421875 disk.rbd2.in‚0 disk.rbd2.out‚ 126.40625 disk.sda2.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚24807.11767578125 network.recv‚893954 disk.rbd3.in‚0 disk.dm-0.in‚ 1724.578125& proc.memory.availableMB‚ 1871.84375# proc.memory.rssMB‚ 2753.60546875 cpu‚25.2905074265222! disk.nvme0c0n1.in‚ 1724.578125( disk.nvme0c0n1.out‚24807.11767578125 network.sent‚ 6.00703e+06 disk.nvme0n1.in‚ 1724.578125% disk./.usageGB‚1106.3297424316406 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# memory_percent‚88.5751724243164 disk.dm-1.out‚99.0625 disk.drbd1777.in‚0 disk.sda.out‚99.0625 disk.sda1.out‚0# disk.dm-0.out‚24807.11767578125* proc.memory.percent‚16.806674003601074 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd3.out‚0~N¾2å- _wandbruntime‚2399  _runtime‚2399[& ÇæÁ _step‚230 & lr-AdamW/pg1‚0.0003661960290025476 & lr-AdamW/pg2‚0.0003661960290025476 trainer/global_step‚2849 % _timestamp‚1.7884399589580264e+09 _runtime‚2399.002743534Dç]ÌçÆ % _timestamp‚1.7884399589580264e+09 _runtime‚2399.002743534 _step‚230 & lr-AdamW/pg1‚0.0003661960290025476 & lr-AdamW/pg2‚0.0003661960290025476 trainer/global_step‚2849æŠÀ2è- _wandbruntime‚2399  _runtime‚2399êÑ}Âú'éôO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271170678976512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316957291700224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311926169734144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46970443 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27210154 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275059418917 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41952801 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41952795 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316957318091776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚322238502049792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53085296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46970893 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95018058 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5272461134253 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271170707110912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884399589588032e+09 _step‚231 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5281210349568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41952801 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53085296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552.Rÿ'êùO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316957291700224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚322238502049792 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317201229153061 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53065263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27210151 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚322238502049792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53085296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46970893 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95018058 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5272461134253 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271170707110912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884399589588032e+09 _step‚231 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5281210349568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41952801 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53085296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271170678976512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316957291700224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311926169734144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46970443 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27210154 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275059418917 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41952801 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41952795 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316957318091776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884399597003682e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283042678272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27219347 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271263067030016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41967031 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317065257797120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46986393 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312032510908416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53083497 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5285642900480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255029363712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276888761793 _step‚232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278518124528128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5274290476921 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283042678272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271263095164416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317306775957369 trainer/global_step‚2849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚322348274083840 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7254985077248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317065231405568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46986808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2399.745490534 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41967037 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53103556 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19767046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚415 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19767458 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41967037 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53103556 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41967031 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53083497 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚322350900697600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5285642900480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceSymÔÚtatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593ELLˆ 'íƒN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41967037 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53103556 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41967031 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53083497 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚322350900697600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5285642900480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚322350900697600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278518052107264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884399597003682e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283042678272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27219347 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271263067030016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41967031 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317065257797120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46986393 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312032510908416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53083497 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5285642900480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255029363712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276888761793 _step‚232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278518124528128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5274290476921 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283042678272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271263095164416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317306775957369 trainer/global_step‚2849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚322348274083840 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7254985077248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317065231405568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46986808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2399.745490534 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41967037 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53103556 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19767046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚415 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19767458èúd0U2î- _wandbruntime‚2399  _runtime‚2399ì¡Ÿ^ïØ 0loss/FBt2/assignment_loss‚0.3981358614012909 _runtime‚2399.746867498 2loss/SRqqt1/assignment_loss‚1.2639313421334277 .loss/FRt3/detection_loss‚2.450947824362162 /loss/FRt4/detection_loss‚2.4019469943913547 /loss/FBt3/detection_loss‚2.1639599353075027 &loss/total_loss‚45.643174850900095 0loss/FRt3/assignment_loss‚4.4254186024536954 1loss/SRqqt4/detection_loss‚1.8254912482655568 epoch‚2 2loss/SRqqt3/assignment_loss‚2.3470310483660017 /loss/FRt1/detection_loss‚1.7157047180665863 /loss/FBt1/detection_loss‚0.6761203674063109 trainer/global_step‚2849 _step‚233 0loss/FRt1/assignment_loss‚1.4755619433191087 0loss/FRt4/assignment_loss‚3.6381889459761707 1loss/SRqqt2/detection_loss‚1.8554192738464246 % _timestamp‚1.7884399597023106e+09 0loss/SRqqt1/detection_loss‚0.963173016783644 0loss/FBt3/assignment_loss‚0.6054570137930568 1loss/FBt4/assignment_loss‚0.18439430291099207 2loss/SRqqt2/assignment_loss‚1.6117533271988997 2loss/SRqqt4/assignment_loss‚1.9133250130259472 1loss/FBt1/assignment_loss‚0.12029591647371633 1loss/SRqqt3/detection_loss‚2.4501645607607707 /loss/FBt2/detection_loss‚2.7663756211598716 -loss/FBt4/detection_loss‚2.53683032308306 /loss/FRt2/detection_loss‚2.5349082866230526 0loss/FRt2/assignment_loss‚3.3186393637914917LÑ¢cðÝ 2loss/SRqqt1/assignment_loss‚1.2639313421334277 .loss/FRt3/detection_loss‚2.450947824362162 /loss/FRt4/detection_loss‚2.4019469943913547 /loss/FBt3/detection_loss‚2.1639599353075027 &loss/total_loss‚45.643174850900095 0loss/FRt3/assignment_loss‚4.4254186024536954 1loss/SRqqt4/detection_loss‚1.8254912482655568 epoch‚2 2loss/SRqqt3/assignment_loss‚2.3470310483660017 /loss/FRt1/detection_loss‚1.7157047180665863 /loss/FBt1/detection_loss‚0.6761203674063109 trainer/global_step‚2849 _step‚233 0loss/FRt1/assignment_loss‚1.4755619433191087 0loss/FRt4/assignment_loss‚3.6381889459761707 1loss/SRqqt2/detection_loss‚1.8554192738464246 % _timestamp‚1.7884399597023106e+09 0loss/SRqqt1/detection_loss‚0.963173016783644 0loss/FBt3/assignment_loss‚0.6054570137930568 1loss/FBt4/assignment_loss‚0.18439430291099207 2loss/SRqqt2/assignment_loss‚1.6117533271988997 2loss/SRqqt4/assignment_loss‚1.9133250130259472 1loss/FBt1/assignment_loss‚0.12029591647371633 1loss/SRqqt3/detection_loss‚2.4501645607607707 /loss/FBt2/detection_loss‚2.7663756211598716 -loss/FBt4/detection_loss‚2.53683032308306 /loss/FRt2/detection_loss‚2.5349082866230526 0loss/FRt2/assignment_loss‚3.3186393637914917 0loss/FBt2/assignment_loss‚0.3981358614012909 _runtime‚2399.746867498éã¡ÿ“ñ‚(:ˆ —ÛåÔ–Ñ¥Ê gpu.0.gpu‚57.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mUÎ#ò‚(:˜ —ÛåÔÒ¸Ë disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1724.578125& disk.nvme0n1.out‚24906.72900390625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24906.72900390625# memory_percent‚88.5765790939331 disk.sda2.out‚0# proc.memory.rssMB‚ 2753.60546875 cpu‚25.288913048687 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 99.67578125 network.sent‚ 6.013237e+06( disk./.usagePercent‚74.2467805209315# disk.dm-0.out‚24906.72900390625 disk.dm-1.in‚0 disk.dm-1.out‚ 99.67578125 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.16357421875 disk.rbd2.out‚ 127.44921875 disk.sda1.in‚0 disk.dm-0.in‚ 1724.578125* proc.memory.percent‚16.806674003601074% disk./.usageGB‚1106.3306846618652 disk.rbd2.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1871.61328125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 99.67578125! disk.nvme0c0n1.in‚ 1724.578125 disk.sda1.out‚0 network.recv‚898469{ž¹“ó‚(:ˆ ¦ÛåÔÊÉ gpu.0.gpu‚59.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚239.491' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8982 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)‰%ô‚(:š ¦ÛåÔÊÙÑË network.recv‚904188 disk.sda.in‚0 disk.sda.out‚ 100.05859375 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 1724.578125( disk.nvme0c0n1.out‚25005.86572265625$ memory_percent‚88.57617378234863 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2753.60546875 disk.sda1.out‚0 disk.dm-0.in‚ 1724.578125# disk.dm-0.out‚25005.86572265625% disk./.usageGB‚1106.3306884765625 disk.rbd1.out‚ 1.2265625 disk.rbd2.out‚ 128.5390625( proc.memory.availableMB‚ 1871.6796875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 100.05859375 disk.nvme0n1.in‚ 1724.578125& disk.nvme0n1.out‚25005.86572265625 disk.rbd0.out‚0 network.sent‚ 6.120955e+06 disk.sda2.in‚0 disk.sda3.out‚ 100.05859375 disk.drbd1777.in‚0* proc.memory.percent‚16.806674003601074) disk./.usagePercent‚74.24678077693903 disk.sda3.in‚0 cpu‚25.287441162546607œT`“õ‚(:ˆ µÛåÔ¨©¯É gpu.0.gpu‚88.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚254.876' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.9752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f¦Óìö‚(:’ µÛåÔË disk.nvme0n1.in‚ 1724.578125$ disk.nvme0n1.out‚25006.576171875 disk.sda3.out‚ 100.51953125! disk.nvme0c0n1.in‚ 1724.578125 network.recv‚908809 disk.sda.in‚0! proc.memory.rssMB‚ 2753.609375* proc.memory.percent‚16.806697845458984 disk.rbd1.out‚ 1.2265625 disk.rbd2.in‚0 disk.sda.out‚ 100.51953125 disk.sda1.in‚0 disk.dm-1.out‚ 100.51953125 cpu‚25.286004429108544$ disk./.usageGB‚1106.330711364746 network.sent‚ 6.126959e+06 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1724.578125& disk.nvme0c0n1.out‚25006.576171875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0$ memory_percent‚88.57817649841309( proc.memory.availableMB‚ 1871.3515625) disk./.usagePercent‚74.24678231298424 disk.rbd2.out‚ 129.58203125! disk.dm-0.out‚25006.576171875 proc.cpu.threads‚17®T?F2÷- _wandbruntime‚2437  _runtime‚2437 qžÉøà ' lr-AdamW/pg1‚0.00036490260144318156 ' lr-AdamW/pg2‚0.00036490260144318156 trainer/global_step‚2899 % _timestamp‚1.7884399978167439e+09 _runtime‚2437.861458451 _step‚234YKÜ‘ÎùÈ _step‚234 ' lr-AdamW/pg1‚0.00036490260144318156 ' lr-AdamW/pg2‚0.00036490260144318156 trainer/global_step‚2899 % _timestamp‚1.7884399978167439e+09 _runtime‚2437.861458451ê, s82ú- _wandbruntime‚2437  _runtime‚2437¬aN!ú'ûôO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322354276967424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283169233548800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42664595 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322354276967424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42664601 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5363941858861 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327729744766464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47771906 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327729744766464 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54000348 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327727118166016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20101952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5375441407488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚2437.862418159 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47771464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275790109786112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322607144878637 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 trainer/global_step‚2899 _step‚235 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5375441407488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283169305982976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322354303358976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7379196196864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317243228447744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884399978175526e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327727118166016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20101513 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5372841198592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322609768591269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96644910 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5372841198592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53980315 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53980315 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27669951 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42664595 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552ZàˆÆ üùO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5372841198592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53980315 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53980315 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27669951 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42664595 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322354276967424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283169233548800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42664595 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322354276967424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42664601 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5363941858861 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327729744766464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47771906 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327729744766464 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54000348 E?Dev¦áp¥_iceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327727118166016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20101952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5375441407488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚2437.862418159 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47771464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275790109786112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322607144878637 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 trainer/global_step‚2899 _step‚235 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5375441407488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283169305982976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322354303358976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7379196196864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317243228447744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884399978175526e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327727118166016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20101513 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5372841198592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322609768591269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96644910 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ëg 9Î2ý- _wandbruntime‚2438  _runtime‚2438.™ìÌ'þþM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27679147 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322462243064320 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5374673527296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5374673527296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 % _timestamp‚1.7884399987752292e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚417 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317349544194048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54018608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5377273749504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275882469705216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 _runtime‚2438.820357245 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7381684973568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42678831 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27679150 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42678831 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20108250 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 trainer/global_step‚2899 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322715315395577 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317349569622016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42678837 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53998549 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283264154678784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327836890200064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54018608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53998549 _step‚236 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7381729260032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96697445 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47787397 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322717939108417 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96697445 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47787397 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322717939108417 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5374673527296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5374673527296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 % _timestamp‚1.7884399987752292e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚417 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317349544194048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54018608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5377273749504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275882469705216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 _runtime‚2438.820357245 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7381684973568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42678831 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27679150 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42678831 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20108250 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 trainer/global_step‚2899 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322715315395577 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317349569622016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42678837 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53998549 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283264154678784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327836890200064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54018608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53998549 _step‚236ìpÔz‹2€- _wandbruntime‚2438  _runtime‚2438ñ`ŠžXÒ 1loss/SRqqt2/detection_loss‚2.0156444155034565 0loss/SRqqt4/detection_loss‚1.853976743088828 epoch‚2 1loss/SRqqt1/assignment_loss‚1.154963565189718 2loss/SRqqt4/assignment_loss‚1.5386475136986486 0loss/SRqqt3/detection_loss‚2.298952593253209 _step‚237 /loss/FRt1/detection_loss‚1.7809595295952425 .loss/FRt4/detection_loss‚2.498536331123776 %loss/total_loss‚46.22800751175073 0loss/FRt2/assignment_loss‚3.7791293744857493 2loss/SRqqt3/assignment_loss‚2.2966593267061772 .loss/FRt2/detection_loss‚2.300126017859349 /loss/FBt3/detection_loss‚2.7979789773623147 $ _timestamp‚1.788439998776925e+09 0loss/FRt3/assignment_loss‚3.8678217178735976 0loss/FRt4/assignment_loss‚3.0950130608346726 2loss/FBt2/assignment_loss‚0.015020642184286512 /loss/FBt1/detection_loss‚0.6360146003348324 -loss/FBt4/detection_loss‚3.80183892590659 1loss/FBt1/assignment_loss‚0.15011881950904715 /loss/FRt3/detection_loss‚2.6934917844258823 1loss/SRqqt1/detection_loss‚1.2712705323570652 .loss/FBt2/detection_loss‚2.496065752846854 /loss/FRt1/assignment_loss‚1.577752102304388 0loss/FBt3/assignment_loss‚0.2464162038328747 0loss/FBt4/assignment_loss‚0.4213893526632871 trainer/global_step‚2899 2loss/SRqqt2/assignment_loss‚1.6402196288108826 _runtime‚2438.821476463†b‹ä]‚× 0loss/FRt2/assignment_loss‚3.7791293744857493 2loss/SRqqt3/assignment_loss‚2.2966593267061772 .loss/FRt2/detection_loss‚2.300126017859349 /loss/FBt3/detection_loss‚2.7979789773623147 $ _timestamp‚1.788439998776925e+09 0loss/FRt3/assignment_loss‚3.8678217178735976 0loss/FRt4/assignment_loss‚3.0950130608346726 2loss/FBt2/assignment_loss‚0.015020642184286512 /loss/FBt1/detection_loss‚0.6360146003348324 -loss/FBt4/detection_loss‚3.80183892590659 1loss/FBt1/assignment_loss‚0.15011881950904715 /loss/FRt3/detection_loss‚2.6934917844258823 1loss/SRqqt1/detection_loss‚1.2712705323570652 .loss/FBt2/detection_loss‚2.496065752846854 /loss/FRt1/assignment_loss‚1.577752102304388 0loss/FBt3/assignment_loss‚0.2464162038328747 0loss/FBt4/assignment_loss‚0.4213893526632871 trainer/global_step‚2899 2loss/SRqqt2/assignment_loss‚1.6402196288108826 _runtime‚2438.821476463 1loss/SRqqt2/detection_loss‚2.0156444155034565 0loss/SRqqt4/detection_loss‚1.853976743088828 epoch‚2 1loss/SRqqt1/assignment_loss‚1.154963565189718 2loss/SRqqt4/assignment_loss‚1.5386475136986486 0loss/SRqqt3/detection_loss‚2.298952593253209 _step‚237 /loss/FRt1/detection_loss‚1.7809595295952425 .loss/FRt4/detection_loss‚2.498536331123776 %loss/total_loss‚46.22800751175073í… >´“ƒ‚(:ˆ ÄÛåÔ³½šÊ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚140.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01þ„‚(:“ ÄÛåÔ‚øêÊ disk.sda3.out‚ 100.94921875! disk.nvme0c0n1.in‚ 1724.578125 disk.dm-1.out‚ 100.94921875 disk.sda2.in‚0 disk.nvme0n1.in‚ 1724.578125$ disk.nvme0n1.out‚25105.677734375$ memory_percent‚88.57669830322266 network.recv‚912887 disk.drbd1777.out‚0 disk.sda.in‚0 disk.sda1.in‚0! disk.dm-0.out‚25105.677734375! disk.rbd2.out‚129.73095703125 disk.rbd3.in‚0& proc.memory.availableMB‚ 1871.59375 disk.dm-1.in‚0 disk.rbd3.out‚0 cpu‚25.284576675717485% disk./.usageGB‚1106.3308906555176 disk.dm-0.in‚ 1724.578125 disk.drbd1777.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.806697845458984 network.sent‚ 6.13276e+06) disk./.usagePercent‚74.24679434533836 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 100.94921875! proc.memory.rssMB‚ 2753.609375& disk.nvme0c0n1.out‚25105.677734375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 proc.cpu.threads‚17 disk.sda3.in‚0&7©¥ž…‚(:“ ÓÛåÔï…É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚272.232' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.446400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IŠ»Ò†‚(:“ ÓÛåÔÈ®Ë disk.sda3.out‚ 101.48046875 disk.drbd1777.out‚0$ memory_percent‚88.57696056365967 network.sent‚ 6.241608e+06 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚919615 disk.rbd3.out‚0# disk.dm-0.out‚25205.16455078125! proc.memory.rssMB‚ 2753.609375 cpu‚25.28305751453415 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚25205.16455078125 disk.dm-1.in‚0 disk.dm-1.out‚ 101.48046875) disk./.usagePercent‚74.24680202556438 disk.sda3.in‚0 disk.drbd1777.in‚0% disk./.usageGB‚1106.3310050964355 disk.sda.out‚ 101.48046875& disk.nvme0n1.out‚25205.16455078125 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.rbd1.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1871.55078125 disk.rbd1.out‚ 1.2265625 disk.sda2.out‚0 disk.nvme0n1.in‚1724.625 disk.nvme0c0n1.in‚1724.625* proc.memory.percent‚16.806697845458984! disk.rbd2.out‚130.77392578125 disk.sda1.in‚0 disk.sda1.out‚0ÝÛ ½‘‡‚(:† âÛåÔõ²¿É gpu.0.gpu‚54.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚268.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã:XÛ"ˆ‚(: âÛåÔÉŠÉË disk.rbd3.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1106.3306045532227 disk.rbd0.out‚0# disk.dm-0.out‚25304.27392578125) proc.memory.availableMB‚ 1871.19140625 network.sent‚ 6.246789e+06) disk./.usagePercent‚74.24677514477328 disk.sda2.in‚0& disk.nvme0n1.out‚25304.27392578125 disk.sda3.out‚ 102.2109375 disk.sda.out‚ 102.2109375 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚1724.625 disk.dm-1.out‚ 102.2109375 disk.rbd1.in‚0# memory_percent‚88.5791540145874* proc.memory.percent‚16.806697845458984 network.recv‚923182! disk.rbd2.out‚131.86376953125 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚25304.27392578125 disk.dm-1.in‚0 cpu‚25.281748320821258;Àù proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚1724.625 disk.rbd1.out‚ 1.2265625! proc.memory.rssMB‚ 2753.609375 disk.nvme0n1.in‚1724.625 disk.rbd0.in‚0 disk.drbd1777.in‚0r:p?2‰- _wandbruntime‚2477  _runtime‚2477•ƒSÈŠÂ ' lr-AdamW/pg1‚0.00036357350740679405 ' lr-AdamW/pg2‚0.00036357350740679405 trainer/global_step‚2949 $ _timestamp‚1.788440037013452e+09 _runtime‚2477.058145066 _step‚238û k9Í‹Ç _runtime‚2477.058145066 _step‚238 ' lr-AdamW/pg1‚0.00036357350740679405 ' lr-AdamW/pg2‚0.00036357350740679405 trainer/global_step‚2949 $ _timestamp‚1.788440037013452e+09îî£Ûà2Œ- _wandbruntime‚2477  _runtime‚2477ŒYš ù'óO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327751262234624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333218360882688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20442192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5455422583469 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2949 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280409512461312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333215734282240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5464472047616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5467072256512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54895367 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54895367 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287915072788480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328015684316845 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322560261733376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328018308029477 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322560287161344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43376401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43376395 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327751262234624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54915400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2477.059052616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333218360882688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788440037014206e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20441746 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096ZÚ¬‚þ'ŽøO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98271762 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7505588461568 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327751262234624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333218360882688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20442192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5455422583469 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2949 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280409512461312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333215734282240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5464472047616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5467072256512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54895367 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54895367 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287915072788480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328015684316845 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322560261733376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328018308029477 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322560287161344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43376401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43376395 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327751262234624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54915400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2477.059052616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333218360882688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788440037014206e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20441746 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5466304376320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322666602907648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43390637 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 _runtime‚2477.800777933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5466304376320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333325506316288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 % _timestamp‚1.7884400377554176e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280501872380416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328126478546625 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5459850211009 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333328132930048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43390631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43390631 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98324297 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54913601 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327859228331520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 trainer/global_step‚2949 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48587460 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327859201939968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54913601 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54933660 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5466304376320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322666602907648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43390637 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 _runtime‚2477.800777933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5466304376320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333325506316288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 % _timestamp‚1.7884400377554176e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280501872380416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328126478546625 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5459850211009 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333328132930048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43390631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43390631 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98324297 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54913601 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327859228331520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 trainer/global_step‚2949 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48587460 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327859201939968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54913601 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98304232 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54933660 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5558703105536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28589551 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55810419 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7632341709824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44088201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44088195 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20783204 _step‚243 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333148273893376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49372758 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333148247501824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327877320446976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55830452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5558703105536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5556102896640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44088201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5546903308077 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327877345874944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338704350398464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28589554 trainer/global_step‚2999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338704350398464 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5556102896640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338706976998912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚2813440061öÜžùO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333426847467685 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 % _timestamp‚1.7884400761523445e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292661301146112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚285028915136512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚285028887002112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5549501592741 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338706976998912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 _runtime‚2516.197152816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44088195 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20783204 _step‚243 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333148273893376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49372758 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333148247501824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327877320446976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55830452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5558703105536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5556102896640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44088201 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5546903308077 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327877345874944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338704350398464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28589554 trainer/global_step‚2999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338704350398464 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5556102896640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338706976998912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55810419 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20782748 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5558703105536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28589551 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55810419 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7632341709824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44088201 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 OEDeviceStatsMonitor.on_train_batch_sC÷'# tart/allocation.small_pool.current‚20033 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55828653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44102431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20789608 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333256213598720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333256187207168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28598747 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5548732650745 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44102437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 _step‚244 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333256213598720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28598750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292756167891456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333532394271993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327983661621248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55828653 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5557935225344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338816749046272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292756240312320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55848712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44102437 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285121303190016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5551330935617 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 trainer/global_step‚2999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7634937122304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884400768848765e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55848712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49388355 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338816749046272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7634892835840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333535017984833 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560535447552 _runtime‚2516.929961424lTž‹ '¡ƒN F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333256213598720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28598750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292756167891456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333532394271993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327983661621248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55828653 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5557935225344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338816749046272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292756240312320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55848712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44102437 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285121303190016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5551330935617 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 trainer/global_step‚2999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7634937122304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884400768848765e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55848712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49388355 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338816749046272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7634892835840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333535017984833 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560535447552 _runtime‚2516.929961424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99951149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20790032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285121275055616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55828653 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44102431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20789608 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333256213598720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333256187207168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28598747 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5548732650745 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44102437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 _step‚244ôÄ9íš2¢- _wandbruntime‚2516  _runtime‚2516!„N/\£Ö /loss/FRt4/assignment_loss‚4.045063836233957 2loss/SRqqt3/assignment_loss‚2.2852436157649962 0loss/FBt4/assignment_loss‚0.2306147398693221 /loss/FRt1/detection_loss‚1.5802052386345402 /loss/FBt1/detection_loss‚0.5488998935419941 /loss/FRt3/assignment_loss‚4.468076730385805 2loss/SRqqt2/assignment_loss‚1.8164726791336485 2loss/SRqqt1/assignment_loss‚0.9507870285407357 0loss/SRqqt4/detection_loss‚2.161762234000933 /loss/FRt1/assignment_loss‚1.658869627021974 2loss/SRqqt4/assignment_loss‚1.6717490759633837 0loss/FBt3/assignment_loss‚0.2639476009644568 1loss/SRqqt2/detection_loss‚1.8125464990735054 0loss/FRt2/assignment_loss‚3.1228857872097993 /loss/FRt2/detection_loss‚2.3461643346520358 /loss/FRt3/detection_loss‚2.2929451007109423 _step‚245 1loss/FBt2/assignment_loss‚0.34859965573996304 /loss/FBt2/detection_loss‚2.8652178406715394 /loss/FBt3/detection_loss‚3.5709420442581177 % _timestamp‚1.7884400768864706e+09 1loss/FBt1/assignment_loss‚0.21646279488894576 /loss/FRt4/detection_loss‚2.5961721553688957 1loss/SRqqt1/detection_loss‚0.9538718954376553 %loss/total_loss‚47.38899785398991 epoch‚2 _runtime‚2516.931021692 1loss/SRqqt3/detection_loss‚2.4618665216297937 .loss/FBt4/detection_loss‚3.119630924292973 trainer/global_step‚2999<[áÞa¤Û 0loss/FRt2/assignment_loss‚3.1228857872097993 /loss/FRt2/detection_loss‚2.3461643346520358 /loss/FRt3/detection_loss‚2.2929451007109423 _step‚245 1loss/FBt2/assignment_loss‚0.34859965573996304 /loss/FBt2/detection_loss‚2.8652178406715394 /loss/FBt3/detection_loss‚3.5709420442581177 % _timestamp‚1.7884400768864706e+09 1loss/FBt1/assignment_loss‚0.21646279488894576 /loss/FRt4/detection_loss‚2.5961721553688957 1loss/SRqqt1/detection_loss‚0.9538718954376553 %loss/total_loss‚47.38899785398991 epoch‚2 _runtime‚2516.931021692 1loss/SRqqt3/detection_loss‚2.4618665216297937 .loss/FBt4/detection_loss‚3.119630924292973 trainer/global_step‚2999 /loss/FRt4/assignment_loss‚4.045063836233957 2loss/SRqqt3/assignment_loss‚2.2852436157649962 0loss/FBt4/assignment_loss‚0.2306147398693221 /loss/FRt1/detection_loss‚1.5802052386345402 /loss/FBt1/detection_loss‚0.5488998935419941 /loss/FRt3/assignment_loss‚4.468076730385805 2loss/SRqqt2/assignment_loss‚1.8164726791336485 2loss/SRqqt1/assignment_loss‚0.9507870285407357 0loss/SRqqt4/detection_loss‚2.161762234000933 /loss/FRt1/assignment_loss‚1.658869627021974 2loss/SRqqt4/assignment_loss‚1.6717490759633837 0loss/FBt3/assignment_loss‚0.2639476009644568 1loss/SRqqt2/detection_loss‚1.8125464990735054õ;­ Œž¥‚(:“ ÜåÔ‹Ÿ©É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚257.593' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.518600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûP=Á¦‚(: ÜåÔÈÜÚË% disk.nvme0n1.out‚25503.3623046875" disk.dm-0.out‚25503.3623046875! disk.rbd2.out‚134.09033203125# disk./.usageGB‚1106.33154296875 disk.dm-1.out‚ 104.04296875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda.in‚0 disk.rbd0.out‚0 network.sent‚ 6.367405e+06 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625 disk.nvme0c0n1.in‚1724.625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.sda2.in‚0 disk.sda.out‚ 104.04296875' proc.memory.availableMB‚ 1870.671875 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚25503.3623046875 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 104.04296875$ memory_percent‚88.58232498168945! proc.memory.rssMB‚ 2753.609375 network.recv‚938388 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.806697845458984 cpu‚25.277454690770938) disk./.usagePercent‚74.24683812262671FS|§‚(:’ žÜåÔö£ŒÉ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚263.049' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.60979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fkqû¨‚(: žÜåÔÐÊÊ network.recv‚944332 disk.nvme0n1.in‚1724.625 disk.nvme0c0n1.in‚1724.625" disk.dm-0.out‚25602.4482421875$ memory_percent‚88.57970237731934 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚74.24683812262671 disk.sda1.in‚0 disk.rbd1.out‚ 1.2265625( proc.memory.availableMB‚ 1871.1015625# disk./.usageGB‚1106.33154296875 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2753.609375 network.sent‚ 6.475346e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.in‚1724.625 cpu‚25.276048368642684' disk.nvme0c0n1.out‚25602.4482421875 disk.dm-1.out‚ 104.43359375 disk.rbd1.in‚0* proc.memory.percent‚16.806697845458984 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚135.2548828125 disk.sda.out‚ 104.43359375% disk.nvme0n1.out‚25602.4482421875 disk.sda3.out‚ 104.43359375ëåƒóž©‚(:“ ­ÜåÔè²³Ê gpu.0.gpu‚89.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.154' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.830799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wTOª‚(:„ ­ÜåÔºÕ¡Ë network.sent‚ 6.481207e+06 disk.sda.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚1724.625$ memory_percent‚88.57977390289307 network.recv‚948473 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.rbd3.out‚0 disk.sda.out‚ 105.21875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1871.08984375* proc.memory.percent‚16.806697845458984) disk./.usagePercent‚74.24684042669452 disk.drbd1777.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚25701.794921875 disk.sda3.out‚ 105.21875& disk.nvme0c0n1.out‚25701.794921875! proc.memory.rssMB‚ 2753.609375 cpu‚25.274815631889904% disk./.usageGB‚1106.3315773010254 disk.nvme0c0n1.in‚1724.625 disk.rbd1.out‚ 1.2265625! disk.dm-0.out‚25701.794921875 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚136.3603515625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 105.21875¦gô2«- _wandbruntime‚2555  _runtime‚2555ªÔð%ǬÁ trainer/global_step‚3049 % _timestamp‚1.7884401150424964e+09 _runtime‚2555.087199564 _step‚246 & lr-AdamW/pg1‚0.0003608094352028933 & lr-AdamW/pg2‚0.0003608094352028933 d˜Ì­Æ & lr-AdamW/pg1‚0.0003608094352028933 & lr-AdamW/pg2‚0.0003608094352028933 trainer/global_step‚3049 % _timestamp‚1.7884401150424964e+09 _runtime‚2555.087199564 _step‚246öš?Ì+2®- _wandbruntime‚2555  _runtime‚2555çåo" ¯øO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44799995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56745504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56725471 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647733745664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338832763193261 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5638384032685 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚289648289677312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceSt «zÜatsMonitor.on_train_batch_start/inactive_split.all.current‚466 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344192966514688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _runtime‚2555.088115269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29049351 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5640982317349 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚338545232769024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338545259160576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884401150432727e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5650333954560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297407314500096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7759071219712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333194379160576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21125491 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333194404588544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚289648317811712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44800001 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338835386905893 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44800001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _step‚247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21125028 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338545259160576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44799995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56745504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56725471 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5647733745664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338832763193261 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5638384032685 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚289648289677312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344192966514688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _runtime‚2555.088115269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29049351 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5640982317349 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚338545232769024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338545259160576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884401150432727e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5650333954560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297407314500096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7759071219712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333194379160576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609÷Oþ2±- _wandbruntime‚2555  _runtime‚2555Éw¾7'²N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44814231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 _step‚248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44814237 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚338653172474368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21131835 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚338653172474368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44814237 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50190815 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338943557423041 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338940933710201 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5652166296576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44814231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚3049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29058547 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56743705 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5649566074368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56743705 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 % _timestamp‚1.7884401157775826e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚338653198865920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56763764 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297502225353728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333300720334848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚338653198865920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50190382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚289740677730816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29058550 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0µÈaų†N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338943557423041 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338940933710201 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5652166296576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44814231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚3049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29058547 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56743705 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5649566074368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56743705 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 % _timestamp‚1.7884401157775826e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚338653198865920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56763764 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297502225353728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333300720334848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚338653198865920 K>DeviceStatsMonitÖùGor.on_train_batch_end/inactive_split.all.freed‚50190382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚289740677730816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29058550 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44814231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 _step‚248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44814237 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚338653172474368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21131835 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚338653172474368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44814237 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50190815 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864øžÖU×2´- _wandbruntime‚2555  _runtime‚2555l ð\µÖ 2loss/SRqqt2/assignment_loss‚1.8442740947008134 0loss/SRqqt2/detection_loss‚2.131908380322986 /loss/FBt1/detection_loss‚0.5021607271544708 /loss/FRt1/detection_loss‚1.5983082485143785 2loss/SRqqt3/assignment_loss‚2.0080690809658597 2loss/SRqqt4/assignment_loss‚1.4451607090811576 /loss/FBt3/assignment_loss‚0.367717618891038 /loss/FBt3/detection_loss‚2.9339606165885925 0loss/FBt4/assignment_loss‚0.1942774567546116 /loss/FRt4/detection_loss‚2.3069350434674156 1loss/SRqqt4/detection_loss‚1.8787542831513189 trainer/global_step‚3049 /loss/FRt2/detection_loss‚2.4990851784294303 /loss/FRt3/assignment_loss‚4.360900072490468 1loss/FBt2/assignment_loss‚0.09844136128335126 /loss/FRt3/detection_loss‚2.6573704831740437 %loss/total_loss‚45.86301181158106 _runtime‚2555.823787763 /loss/FRt2/assignment_loss‚3.396211499517614 2loss/SRqqt1/assignment_loss‚1.0185411921344125 1loss/FBt1/assignment_loss‚0.15133627948261552 /loss/FBt2/detection_loss‚3.1430670022964478 % _timestamp‚1.7884401157791338e+09 _step‚249 epoch‚2 /loss/FRt1/assignment_loss‚1.703825785054101 0loss/FRt4/assignment_loss‚3.2581430355707806 1loss/SRqqt3/detection_loss‚2.3273350732667106 1loss/SRqqt1/detection_loss‚1.3378529333724425 /loss/FBt4/detection_loss‚2.6993756559160023ù ca¶Û /loss/FRt1/detection_loss‚1.5983082485143785 2loss/SRqqt3/assignment_loss‚2.0080690809658597 2loss/SRqqt4/assignment_loss‚1.4451607090811576 /loss/FBt3/assignment_loss‚0.367717618891038 /loss/FBt3/detection_loss‚2.9339606165885925 0loss/FBt4/assignment_loss‚0.1942774567546116 /loss/FRt4/detection_loss‚2.3069350434674156 1loss/SRqqt4/detection_loss‚1.8787542831513189 trainer/global_step‚3049 /loss/FRt2/detection_loss‚2.4990851784294303 /loss/FRt3/assignment_loss‚4.360900072490468 1loss/FBt2/assignment_loss‚0.09844136128335126 /loss/FRt3/detection_loss‚2.6573704831740437 %loss/total_loss‚45.86301181158106 _runtime‚2555.823787763 /loss/FRt2/assignment_loss‚3.396211499517614 2loss/SRqqt1/assignment_loss‚1.0185411921344125 1loss/FBt1/assignment_loss‚0.15133627948261552 /loss/FBt2/detection_loss‚3.1430670022964478 % _timestamp‚1.7884401157791338e+09 _step‚249 epoch‚2 /loss/FRt1/assignment_loss‚1.703825785054101 0loss/FRt4/assignment_loss‚3.2581430355707806 1loss/SRqqt3/detection_loss‚2.3273350732667106 1loss/SRqqt1/detection_loss‚1.3378529333724425 /loss/FBt4/detection_loss‚2.6993756559160023 2loss/SRqqt2/assignment_loss‚1.8442740947008134 0loss/SRqqt2/detection_loss‚2.131908380322986 /loss/FBt1/detection_loss‚0.5021607271544708ùWež·‚(:“ ¼ÜåÔêêžÉ gpu.0.gpu‚74.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚264.716' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.943200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZŠŒ¸‚(:Š ¼ÜåÔäñ„Ì disk.nvme0n1.in‚1724.625 disk.rbd0.out‚0 disk.dm-1.out‚ 105.80078125( proc.memory.availableMB‚ 1870.8671875% disk./.usageGB‚1106.3317070007324 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.nvme0c0n1.in‚1724.625 disk.dm-0.out‚25702.33203125# disk.nvme0n1.out‚25702.33203125 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0! disk.rbd2.out‚137.58740234375! proc.memory.rssMB‚ 2753.609375) disk./.usagePercent‚74.24684913095068 disk.rbd2.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.806697845458984 disk.sda3.out‚ 105.80078125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 network.recv‚952865 disk.rbd3.out‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚25702.33203125 disk.sda.in‚0 disk.sda2.out‚0 cpu‚25.273499065929283 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 6.487046e+06 disk.rbd3.in‚0 disk.sda.out‚ 105.80078125 disk.sda2.in‚0$ memory_percent‚88.58113288879395ƒñ1T¹‚(:’ ËÜåÔÞ±šÉ gpu.0.gpu‚70.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚253.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.714000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*º‚(:’ ËÜåÔ±™ Ë disk.dm-1.in‚0 disk.rbd1.out‚ 1.2265625 disk.rbd0.in‚0 disk.drbd1777.in‚0! disk.rbd2.out‚138.63037109375 disk.sda.out‚ 106.68359375 disk.sda2.in‚0 disk.sda3.out‚ 106.68359375 disk.rbd0.out‚0# disk.dm-0.out‚25802.12060546875 disk.dm-1.out‚ 106.68359375 disk.drbd1777.out‚0 cpu‚25.27213278623285 network.recv‚959599) disk./.usagePercent‚74.24684913095068( disk.nvme0c0n1.out‚25802.12060546875 disk.sda1.in‚0 disk.nvme0n1.in‚1724.625 network.sent‚ 6.596069e+06% disk./.usageGB‚1106.3317070007324 disk.nvme0c0n1.in‚1724.625 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚1724.625 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚88.58170509338379( proc.memory.availableMB‚ 1870.7734375 disk.sda3.in‚0 disk.sda.in‚0! proc.memory.rssMB‚ 2753.609375* proc.memory.percent‚16.806697845458984 disk.rbd2.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚25802.12060546875 proc.cpu.threads‚17j3å2»- _wandbruntime‚2594  _runtime‚2594ÙÍ£rɼà ' lr-AdamW/pg2‚0.00035937503306388425 trainer/global_step‚3099 % _timestamp‚1.7884401542040474e+09 _runtime‚2594.248769204 _step‚250 ' lr-AdamW/pg1‚0.00035937503306388425ëÖe–Î½È ' lr-AdamW/pg2‚0.00035937503306388425 trainer/global_step‚3099 % _timestamp‚1.7884401542040474e+09 _runtime‚2594.248769204 _step‚250 ' lr-AdamW/pg1‚0.00035937503306388425úöª­:2¾- _wandbruntime‚2594  _runtime‚2594ƒÏGý'¿÷O O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344243926344101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57660556 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50976898 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344241302631469 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302153023080448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7885302593536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 trainer/global_step‚3099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343942244427776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21468212 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45511795 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5732463041957 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 _step‚251 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349681582630912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45511801 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349684209231360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5741964803584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5729864757293 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57660556 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57640523 _runtime‚2594.249937476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7885256196608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50977366 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21467747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5739364594688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5739364594688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29509154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29509151 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349684209231360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338511437874176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343942244427776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103152318 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343942218036224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302152948549120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294267692352512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338511463302144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 $ _timestamp‚1.788440154204906e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5741964803584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349681582630912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45511795 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343942218036224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45511801 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349684209231360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5741964803584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5729864757293 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57660556 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57640523 _runtime‚2594.249937476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7885256196608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50977366 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21467747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5739364594688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5739364594688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29509154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29509151 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349684209231360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338511437874176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343942244427776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103152318 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343942218036224 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302152948549120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294267692352512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338511463302144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 $ _timestamp‚1.788440154204906e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5741964803584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349681582630912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45511795 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343942218036224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50992860 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21474969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21474513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45526031 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302247858018816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294360108540416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57678816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5743797145600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302247930439680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7887777612800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57658757 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349793981278720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344349473148409 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45526031 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚344352096861249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338617779048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884401549381678e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344050184133120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344050157741568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29518347 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2594.983219542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349793981278720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344349473148409 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45526031 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚344352096861249 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338617779048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884401549381678e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344050184133120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344050157741568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29518347 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2594.983219542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50992860 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21474969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21474513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45526031 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302247858018816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294360108540416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103184788 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57678816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5743797145600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302247930439680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7887777612800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57658757 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46223595 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46223601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21808785 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349652465782309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58575608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51777739 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29968951 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104779170 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5821345481901 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58575608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚349339203303424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343828522015744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 _runtime‚2633.117950502 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355170198747136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355172825347584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46223595 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306898618176000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚349339203303424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5833595652608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚3149 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5833595652608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚349649842069677$ïr}(ÒýO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚3149 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5833595652608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚349649842069677 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5830995443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298887095027712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46223595 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46223601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21808785 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349652465782309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58575608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51777739 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29968951 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104779170 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5821345481901 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58575608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚349339203303424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343828522015744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 _runtime‚2633.117950502 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355170198747136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355172825347584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46223595 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306898618176000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚349339203303424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5833595652608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349447169400320 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51793214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5835427994624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298979483081216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5825773109441 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5823174824569 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349760636299457 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5835427994624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349758012586617 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 $ _timestamp‚1.788440193805765e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46237837 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5832827772416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349447143008768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8014059056640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355282597394944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355279970781184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58573809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306993614558720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306993542137856 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58593868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46237831 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29978147 _runtime‚2633.850838178 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 trainer/global_step‚3149þg8¢îÕ†N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5835427994624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349758012586617 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 $ _timestamp‚1.788440193805765e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46237837 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5832827772416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349447143008768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8014059056640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355282597394944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355279970781184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58573809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306993614558720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306993542137856 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58593868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46237831 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29978147 _runtime‚2633.850838178 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 trainer/global_step‚3149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349447143008768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355282597394944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343934837762048 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44286464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29978150 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5832827772416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298979511215616 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349447169400320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46237837 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343934863190016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [GDeviceStatsMonû¾F¶ itor.on_train_batch_end/active_bytes.large_pool.allocated‚349447169400320 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51793214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5835427994624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298979483081216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5825773109441 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5823174824569 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349760636299457€ÌÂï2Ö- _wandbruntime‚2633  _runtime‚2633øŒT×Î /loss/FBt3/detection_loss‚2.2136313915252686 _runtime‚2633.852251771 /loss/FRt4/assignment_loss‚3.398946974765171 .loss/FRt4/detection_loss‚2.471052963625301 0loss/SRqqt3/detection_loss‚2.434766972387159 .loss/FBt4/detection_loss‚3.244062066078186 epoch‚2 _step‚257 /loss/FBt1/detection_loss‚0.5109962781265747 %loss/total_loss‚45.81771851591981 2loss/SRqqt1/assignment_loss‚0.9713561550503776 /loss/FRt1/detection_loss‚1.5919999379445524 .loss/FRt3/detection_loss‚2.435265067219734 1loss/SRqqt1/detection_loss‚1.0660224371860105 1loss/SRqqt4/detection_loss‚1.7835091755918737 .loss/FBt2/detection_loss‚2.624606430530548 0loss/FRt1/assignment_loss‚1.3736539085706074 /loss/FRt2/assignment_loss‚4.030039614957312 .loss/FBt2/assignment_loss‚0.55641961209476 .loss/FRt2/detection_loss‚2.519756891805193 /loss/FRt3/assignment_loss‚4.149299547076225 2loss/SRqqt4/assignment_loss‚1.5251805415412165 1loss/SRqqt2/assignment_loss‚1.860499443584367 2loss/SRqqt3/assignment_loss‚1.8276128148710407 1loss/FBt1/assignment_loss‚0.20712534748209904 1loss/FBt3/assignment_loss‚0.41411840433085506 0loss/FBt4/assignment_loss‚0.8305477008223534 0loss/SRqqt2/detection_loss‚1.777248838753031 trainer/global_step‚3149 % _timestamp‚1.7884401938076591e+09Þyö!YØÓ /loss/FRt2/assignment_loss‚4.030039614957312 .loss/FBt2/assignment_loss‚0.55641961209476 .loss/FRt2/detection_loss‚2.519756891805193 /loss/FRt3/assignment_loss‚4.149299547076225 2loss/SRqqt4/assignment_loss‚1.5251805415412165 1loss/SRqqt2/assignment_loss‚1.860499443584367 2loss/SRqqt3/assignment_loss‚1.8276128148710407 1loss/FBt1/assignment_loss‚0.20712534748209904 1loss/FBt3/assignment_loss‚0.41411840433085506 0loss/FBt4/assignment_loss‚0.8305477008223534 0loss/SRqqt2/detection_loss‚1.777248838753031 trainer/global_step‚3149 % _timestamp‚1.7884401938076591e+09 /loss/FBt3/detection_loss‚2.2136313915252686 _runtime‚2633.852251771 /loss/FRt4/assignment_loss‚3.398946974765171 .loss/FRt4/detection_loss‚2.471052963625301 0loss/SRqqt3/detection_loss‚2.434766972387159 .loss/FBt4/detection_loss‚3.244062066078186 epoch‚2 _step‚257 /loss/FBt1/detection_loss‚0.5109962781265747 %loss/total_loss‚45.81771851591981 2loss/SRqqt1/assignment_loss‚0.9713561550503776 /loss/FRt1/detection_loss‚1.5919999379445524 .loss/FRt3/detection_loss‚2.435265067219734 1loss/SRqqt1/detection_loss‚1.0660224371860105 1loss/SRqqt4/detection_loss‚1.7835091755918737 .loss/FBt2/detection_loss‚2.624606430530548 0loss/FRt1/assignment_loss‚1.3736539085706074·„½“Ù‚(:ˆ ‡ÝåÔ¶Š˜É gpu.0.gpu‚70.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚260.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¬^Ú‚(:Ž ‡ÝåÔÙÀ“Ë% disk./.usageGB‚1106.3310165405273 disk.rbd1.in‚0 disk.sda.out‚ 108.94140625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚1724.625" disk.dm-0.out‚26100.1669921875 disk.dm-1.out‚ 108.94140625 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 cpu‚25.267137331617278 disk.rbd3.in‚0' disk.nvme0c0n1.out‚26100.1669921875 network.sent‚ 6.721931e+06 network.recv‚978475 disk.sda.in‚0) disk./.usagePercent‚74.24680279358698 disk.drbd1777.out‚0% disk.nvme0n1.out‚26100.1669921875 disk.dm-1.in‚0 disk.drbd1777.in‚0$ memory_percent‚88.58227729797363* proc.memory.percent‚16.806697845458984 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 108.94140625( proc.memory.availableMB‚ 1870.6796875 disk.sda1.in‚0 disk.rbd1.out‚ 1.2265625 disk.rbd2.in‚0 disk.rbd2.out‚ 142.033203125 disk.sda2.out‚0! proc.memory.rssMB‚ 2753.609375 proc.cpu.threads‚17M ¨žÛ‚(:“ –ÝåÔõЩÉ gpu.0.gpu‚71.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚157.953' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.590600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚=ЪÜ‚(:† –ÝåÔ¡·ØÊ disk.rbd3.in‚0& disk.nvme0n1.out‚26199.28466796875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625 disk.rbd2.in‚0 disk.sda1.in‚0$ memory_percent‚88.58280181884766& proc.memory.availableMB‚ 1870.59375 disk.rbd2.out‚ 143.193359375 network.sent‚ 6.829647e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 109.53125) disk./.usagePercent‚74.24680356160958% disk./.usageGB‚1106.3310279846191 disk.rbd3.out‚0# disk.dm-0.out‚26199.28466796875 disk.sda.out‚ 109.53125! proc.memory.rssMB‚ 2753.609375* proc.memory.percent‚16.806697845458984 disk.nvme0n1.in‚1724.625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚26199.28466796875 disk.dm-1.out‚ 109.53125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.recv‚984484 disk.dm-0.in‚1724.625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda.in‚0 cpu‚25.265931826366835«{%d“Ý‚(:ˆ ¥ÝåÔÑäšÉ gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚117.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ-õÞ‚(:ˆ ¥ÝåÔ…ª•Ë cpu‚25.26462885064707% disk./.usageGB‚1106.3310279846191 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 6.835707e+06 disk.drbd1777.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚26299.27978515625 disk.rbd0.in‚0 disk.dm-1.out‚ 110.03125) proc.memory.availableMB‚ 1870.83984375 disk.sda.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2753.609375 disk.rbd2.out‚ 144.240234375 disk.sda.out‚ 110.03125 disk.sda3.out‚ 110.03125 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0$ memory_percent‚88.58129978179932 disk.nvme0c0n1.in‚1724.625( disk.nvme0c0n1.out‚26299.27978515625 disk.dm-0.in‚1724.625# disk.dm-0.out‚26299.27978515625 network.recv‚988835) disk./.usagePercent‚74.24680356160958 disk.rbd2.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.2265625 disk.rbd0.out‚0 disk.nvme0n1.in‚1724.625 disk.sda3.in‚0* proc.memory.percent‚16.806697845458984®J¥K2ß- _wandbruntime‚2672  _runtime‚2672èçaCÉàà trainer/global_step‚3199 % _timestamp‚1.7884402321636646e+09 _runtime‚2672.208326916 _step‚258 ' lr-AdamW/pg1‚0.00035640299128573223 ' lr-AdamW/pg2‚0.00035640299128573223\lZRÎáÈ ' lr-AdamW/pg1‚0.00035640299128573223 ' lr-AdamW/pg2‚0.00035640299128573223 trainer/global_step‚3199 % _timestamp‚1.7884402321636646e+09 _runtime‚2672.208326916 _step‚258‚êÜXG2â- _wandbruntime‚2672  _runtime‚2672wK’Öþ'ãøO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5925226501632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46935401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8137611439616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303506525837312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46935395 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5922626292736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349145580729344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5922626292736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚355058381507885 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22149443 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5915424491173 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59470627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303506497702912 _runtime‚2672.209318314 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚349145555301376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46935401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8137562945536 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _step‚259 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354736214962176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360658814863360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 trainer/global_step‚3199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52577746 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46935395 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311644060648448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354736188570624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360658814863360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354736188570624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30428751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360661441463808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106426061 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59490660 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52578197 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494080 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354736214962176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360661441463808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22148995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311644137276928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚355061005220517 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039#Ê5 (äýO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360661441463808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22148995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311644137276928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚355061005220517 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5925226501632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46935401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8137611439616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303506525837312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46935395 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5922626292736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349145580729344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5922626292736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚355058381507885 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22149443 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5915424491173 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59470627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303506497702912 _runtime‚2672.209318314 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚349145555301376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46935401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8137562945536 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _step‚259 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354736214962176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360658814863360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 trainer/global_step‚3199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52577746 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46935395 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311644060648448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354736188570624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360658814863360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354736188570624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30428751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360661441463808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106426061 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59490660 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52578197 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494080 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354736214962176ƒ2]2å- _wandbruntime‚2672  _runtime‚2672LFçVæN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5927058843648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52593704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355169175737665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311738968805376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46949637 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59508920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360771213511168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46949631 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59488861 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8140129432576 _step‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355166552024825 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5914655549177 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46949631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355166552024825 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5914655549177 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46949631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52593704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355169175737665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311738968805376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46949637 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59508920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144„“é~#2è- _wandbruntime‚2672  _runtime‚2672[¹:žWéÑ /loss/FRt3/assignment_loss‚4.264000259912931 1loss/FBt1/assignment_loss‚0.09989514119820654 0loss/FBt2/assignment_loss‚0.2299052123978202 1loss/SRqqt3/detection_loss‚2.6063771482024873 1loss/SRqqt4/detection_loss‚1.8017270625271695 %loss/total_loss‚43.49057954310021 0loss/FRt1/assignment_loss‚1.2630921442593848 2loss/SRqqt2/assignment_loss‚1.8808227369473094 0loss/FBt4/assignment_loss‚0.2469989387318492 /loss/FBt1/detection_loss‚0.5852734122347945 /loss/FBt2/detection_loss‚2.7998267241886685 _runtime‚2672.950135615 _step‚261 /loss/FRt2/assignment_loss‚2.580107311992084 2loss/SRqqt3/assignment_loss‚1.4975342473813467 % _timestamp‚1.7884402329051824e+09 2loss/SRqqt4/assignment_loss‚1.8140097558498383 /loss/FBt4/detection_loss‚2.4884824007749557 0loss/FBt3/assignment_loss‚0.2903064058627933 /loss/FRt3/detection_loss‚2.4667514127034407 1loss/SRqqt1/detection_loss‚1.1322788355105065 epoch‚2 1loss/SRqqt1/assignment_loss‚1.002591392627972 .loss/FRt4/detection_loss‚2.606647450672953 ,loss/FRt2/detection_loss‚2.7740658223629 /loss/FBt3/detection_loss‚2.3078983575105667 0loss/FRt4/assignment_loss‚3.0796693783057365 /loss/FRt1/detection_loss‚1.5948152914643288 /loss/SRqqt2/detection_loss‚2.07750269948017 trainer/global_step‚3199u¢R!\êÖ /loss/FRt3/detection_loss‚2.4667514127034407 1loss/SRqqt1/detection_loss‚1.1322788355105065 epoch‚2 1loss/SRqqt1/assignment_loss‚1.002591392627972 .loss/FRt4/detection_loss‚2.606647450672953 ,loss/FRt2/detection_loss‚2.7740658223629 /loss/FBt3/detection_loss‚2.3078983575105667 0loss/FRt4/assignment_loss‚3.0796693783057365 /loss/FRt1/detection_loss‚1.5948152914643288 /loss/SRqqt2/detection_loss‚2.07750269948017 trainer/global_step‚3199 /loss/FRt3/assignment_loss‚4.264000259912931 1loss/FBt1/assignment_loss‚0.09989514119820654 0loss/FBt2/assignment_loss‚0.2299052123978202 1loss/SRqqt3/detection_loss‚2.6063771482024873 1loss/SRqqt4/detection_loss‚1.8017270625271695 %loss/total_loss‚43.49057954310021 0loss/FRt1/assignment_loss‚1.2630921442593848 2loss/SRqqt2/assignment_loss‚1.8808227369473094 0loss/FBt4/assignment_loss‚0.2469989387318492 /loss/FBt1/detection_loss‚0.5852734122347945 /loss/FBt2/detection_loss‚2.7998267241886685 _runtime‚2672.950135615 _step‚261 /loss/FRt2/assignment_loss‚2.580107311992084 2loss/SRqqt3/assignment_loss‚1.4975342473813467 % _timestamp‚1.7884402329051824e+09 2loss/SRqqt4/assignment_loss‚1.8140097558498383 /loss/FBt4/detection_loss‚2.4884824007749557 0loss/FBt3/assignment_loss‚0.2903064058627933…QsÑ“ë‚(:ˆ ´ÝåÔñá–É gpu.0.gpu‚70.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚267.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aòíì‚(:† ´ÝåÔåÈ„Ì disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚1724.625 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda.out‚ 110.59375 disk.rbd0.out‚0$ memory_percent‚88.58294486999512 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 110.59375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26398.40380859375 cpu‚25.2634250697689 disk.rbd2.out‚ 145.392578125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚1724.625 network.recv‚993054% disk./.usageGB‚1106.3310279846191& disk.nvme0n1.out‚26398.40380859375 disk.dm-1.out‚ 110.59375( proc.memory.availableMB‚ 1870.5703125! proc.memory.rssMB‚ 2753.609375* proc.memory.percent‚16.806697845458984 network.sent‚ 6.841429e+06# disk.dm-0.out‚26398.40380859375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚1724.625 disk.sda3.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚74.24680356160958 disk.sda2.out‚0 disk.dm-1.in‚0äEõží‚(:“ ÃÝåÔí¤ˆÉ gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.163' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.232600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ªî‚(:‘ ÃÝåÔ–Æ¢Ë# disk.dm-0.out‚26501.96240234375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 network.sent‚ 6.949498e+06 disk.dm-0.in‚1724.625 disk.drbd1777.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚26501.96240234375 cpu‚25.262265590006805 disk.rbd0.in‚0 network.recv‚998764 disk.rbd1.out‚ 1.2265625 disk.sda.out‚ 111.18359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚1724.625( disk.nvme0c0n1.out‚26501.96240234375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚1724.625# memory_percent‚88.5814905166626 disk.rbd2.out‚ 146.439453125 disk.dm-1.in‚0 disk.dm-1.out‚ 111.18359375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 111.18359375% disk./.usageGB‚1105.1003379821777* proc.memory.percent‚16.806697845458984 disk.rbd2.in‚0) disk./.usagePercent‚74.16421092292805) proc.memory.availableMB‚ 1870.80859375 disk.rbd0.out‚0! proc.memory.rssMB‚ 2753.609375¯˜9¹2ï- _wandbruntime‚2711  _runtime‚2711ã@qÚÆðÀ trainer/global_step‚3249 % _timestamp‚1.7884402711774282e+09 _runtime‚ 2711.22213384 _step‚262 & lr-AdamW/pg1‚0.0003548659710156412 & lr-AdamW/pg2‚0.0003548659710156412¡‘¬±ËñÅ & lr-AdamW/pg2‚0.0003548659710156412 trainer/global_step‚3249 % _timestamp‚1.7884402711774282e+09 _runtime‚ 2711.22213384 _step‚262 & lr-AdamW/pg1‚0.0003548659710156412†P°Ê_2ò- _wandbruntime‚2711  _runtime‚2711j)O©!óøO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22489925 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 _runtime‚2711.223113846 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360133200229376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366150057580032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366150057580032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108032874 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366147430979584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60385679 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47647195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6006905215781 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6016857350656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60405712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47647195 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30888551 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8263569497088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360133173837824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6016857350656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60405712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6004306931117 _step‚263 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30888554 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308125900378112 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60385679 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360133200229376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6014257141760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308125928512512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_poÂRUol.peak‚ 2338332672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366147430979584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47647201 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53378027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360469544658725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316389423478272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 trainer/global_step‚3249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360466920946093 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354462639442944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0d¤jˆ(ôýO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6014257141760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354462614014976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47647195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6006905215781 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6016857350656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60405712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47647195 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30888551 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8263569497088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360133173837824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6016857350656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60405712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6004306931117 _step‚263 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30888554 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308125900378112 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60385679 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360133200229376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6014257141760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308125928512512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366147430979584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47647201 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53378027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360469544658725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316389423478272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 trainer/global_step‚3249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360466920946093 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354462639442944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22489925 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 _runtime‚2711.223113846 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360133200229376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366150057580032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366150057580032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108032874 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366147430979584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60385679 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8266083924992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47661431 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6018689692672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354568980617216 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108085409 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6016089470464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 _runtime‚2711.962520571 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360575091463033 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30897747 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308218316566016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8266035444224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53393921 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22496622 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6008734558657 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6018689692672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30897750 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360241139934720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788440271917391e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316484400491008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360577715175873 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360241113543168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316484323875840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573z[µK '÷†N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366259829627392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60423972 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53393921 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22496622 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6008734558657 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6018689692672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30897750 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360241139934720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788440271917391e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316484400491008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360577715175873 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚264 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360241113543168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316484323875840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47661431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22496174 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6006136273785 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308218288431616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚3249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8266083924992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47661431 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6018689692672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354568980617216 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108085409 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6016089470464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 _runtime‚2711.962520571 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360575091463033 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30897747 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308218316566016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8266035444224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31348351 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22829013 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6095787655725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365530185496576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8389673850368 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61300731 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚267 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61320764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359779672728576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48359001 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371638673696256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6105887990784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365875460384301 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359779698156544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48358995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312745331187712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365878084096933 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61300731 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371636047095808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61300731 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚267 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61320764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359779672728576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48359001 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371638673696256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6105887990784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72434176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365875460384301 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359779698156544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48358995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312745331187712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365878084096933 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61300731 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371636047095808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320‹^˜¶2‡- _wandbruntime‚2751  _runtime‚27512À; ˆ‚N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321229944982016 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 trainer/global_step‚3299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61318965 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8392225740800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48373237 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109692196 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22835731 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6097616998393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371748445743616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6110320541696 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100215283265 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6107720319488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312837691106816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚268 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54193278 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31357547 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K;DeviceStaÉ+ÀhtsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6110320541696 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54193716 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61318965 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359886013902848 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109712261 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365638098810368 _runtime‚2751.242423393 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48373231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365638125201920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365986254614081 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22836166 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31357550 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321229870464000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8392179357184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61339024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371748445743616 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48373237 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359886039330816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371745819129856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552üùC0 '‰‡N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61318965 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359886013902848 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109712261 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365638098810368 _runtime‚2751.242423393 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48373231 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365638125201920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365986254614081 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22836166 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31357550 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321229870464000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8392179357184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61339024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371748445743616 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48373237 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359886039330816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371745819129856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321229944982016 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 trainer/global_step‚3299 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61318965 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8392225740800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48373237 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109692196 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22835731 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6097616998393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371748445743616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6110320541696 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100215283265 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6107720319488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312837691106816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚268 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54193278 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31357547 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6110320541696 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54193716 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647ŒÜéÔÈ2Š- _wandbruntime‚2751  _runtime‚2751ˆˆèZ‹Ô /loss/FBt1/detection_loss‚0.6298596242810864 0loss/FRt4/assignment_loss‚3.0048159586058723 0loss/SRqqt2/detection_loss‚1.711966760456562 1loss/SRqqt3/detection_loss‚2.4756474129615293 /loss/FBt3/detection_loss‚2.0860659100792627 epoch‚2 2loss/SRqqt4/assignment_loss‚1.6563019124316234 .loss/FRt3/detection_loss‚2.430709986125725 /loss/FBt2/detection_loss‚2.8054297783158044 .loss/FRt2/detection_loss‚2.150956374075678 % _timestamp‚1.7884403111995306e+09 0loss/FRt1/assignment_loss‚1.4684902088982719 1loss/FBt1/assignment_loss‚0.08311883288908795 %loss/total_loss‚43.74078099560538 trainer/global_step‚3299 1loss/FBt2/assignment_loss‚0.17667490707456387 0loss/FBt4/assignment_loss‚0.5001572014763951 1loss/SRqqt1/detection_loss‚0.8110317527383998 1loss/SRqqt4/detection_loss‚2.0270270047818912 _step‚269 2loss/SRqqt2/assignment_loss‚1.5626268575101026 0loss/FBt3/assignment_loss‚0.2501618208956312 /loss/FBt4/detection_loss‚2.5531323552131653 _runtime‚2751.244185864 /loss/FRt2/assignment_loss‚3.186009027339794 /loss/FRt3/assignment_loss‚4.563866944874034 1loss/SRqqt3/assignment_loss‚2.187103797110819 /loss/FRt1/detection_loss‚1.8143621566040176 .loss/FRt4/detection_loss‚2.494110706117418 2loss/SRqqt1/assignment_loss‚1.1111537047486375ö F_ŒÙ .loss/FRt3/detection_loss‚2.430709986125725 /loss/FBt2/detection_loss‚2.8054297783158044 .loss/FRt2/detection_loss‚2.150956374075678 % _timestamp‚1.7884403111995306e+09 0loss/FRt1/assignment_loss‚1.4684902088982719 1loss/FBt1/assignment_loss‚0.08311883288908795 %loss/total_loss‚43.74078099560538 trainer/global_step‚3299 1loss/FBt2/assignment_loss‚0.17667490707456387 0loss/FBt4/assignment_loss‚0.5001572014763951 1loss/SRqqt1/detection_loss‚0.8110317527383998 1loss/SRqqt4/detection_loss‚2.0270270047818912 _step‚269 2loss/SRqqt2/assignment_loss‚1.5626268575101026 0loss/FBt3/assignment_loss‚0.2501618208956312 /loss/FBt4/detection_loss‚2.5531323552131653 _runtime‚2751.244185864 /loss/FRt2/assignment_loss‚3.186009027339794 /loss/FRt3/assignment_loss‚4.563866944874034 1loss/SRqqt3/assignment_loss‚2.187103797110819 /loss/FRt1/detection_loss‚1.8143621566040176 .loss/FRt4/detection_loss‚2.494110706117418 2loss/SRqqt1/assignment_loss‚1.1111537047486375 /loss/FBt1/detection_loss‚0.6298596242810864 0loss/FRt4/assignment_loss‚3.0048159586058723 0loss/SRqqt2/detection_loss‚1.711966760456562 1loss/SRqqt3/detection_loss‚2.4756474129615293 /loss/FBt3/detection_loss‚2.0860659100792627 epoch‚2 2loss/SRqqt4/assignment_loss‚1.6563019124316234Kà-V“‚(:ˆ ÿÝåÔª¤É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óº‚Ž‚(:Œ ÿÝåÔ‡ô˜Ì disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚1724.625) disk./.usagePercent‚74.18093128700434% disk./.usageGB‚1105.3494834899902 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1870.44140625 network.sent‚ 7.075206e+06 disk.dm-1.out‚113.4375 disk.drbd1777.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚26757.68994140625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625 disk.drbd1777.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚26757.68994140625$ memory_percent‚88.58373165130615# disk.dm-0.out‚26757.68994140625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚113.4375 disk.rbd0.out‚0 network.recv‚ 1.017598e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 150.830078125 disk.dm-0.in‚1724.625! proc.memory.rssMB‚ 2753.609375* proc.memory.percent‚16.806697845458984 cpu‚25.257588808123952 proc.cpu.threads‚17 disk.sda.out‚113.4375 disk.rbd1.out‚ 1.2265625€ßŽÇ“‚(:ˆ ŽÞåÔ¾²ÿÈ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚168.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09á~$‚(:™ ŽÞåÔ£“ÄÊ cpu‚25.256419276799665 disk.sda2.out‚0# disk.dm-0.out‚26757.99072265625 disk.dm-1.out‚ 113.76171875* proc.memory.percent‚16.806697845458984 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 113.76171875 proc.cpu.threads‚17! disk.rbd2.out‚152.10791015625 disk.nvme0c0n1.in‚1724.625( disk.nvme0c0n1.out‚26757.99072265625 disk.sda.in‚0 network.sent‚ 7.183002e+06 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0# memory_percent‚88.5845422744751) proc.memory.availableMB‚ 1870.30859375! proc.memory.rssMB‚ 2753.609375 disk.sda.out‚ 113.76171875 disk.sda1.out‚0% disk./.usageGB‚1105.3486366271973 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.2265625 network.recv‚ 1.023284e+06) disk./.usagePercent‚74.18087445333174 disk.nvme0n1.in‚1724.625& disk.nvme0n1.out‚26757.99072265625 disk.dm-0.in‚1724.625Uj52‘- _wandbruntime‚2789  _runtime‚27899¸ÅÌÇ’Á % _timestamp‚1.7884403498527033e+09 _runtime‚2789.897340293 _step‚270 & lr-AdamW/pg1‚0.0003516915332102792 & lr-AdamW/pg2‚0.0003516915332102792 trainer/global_step‚3349ˆ>i°Ì“Æ _runtime‚2789.897340293 _step‚270 & lr-AdamW/pg1‚0.0003516915332102792 & lr-AdamW/pg2‚0.0003516915332102792 trainer/global_step‚3349 % _timestamp‚1.7884403498527033e+09ŽÉÁê2”- _wandbruntime‚2789  _runtime‚2789=C¥ þ'•øO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49070801 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377124663212032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49070795 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325880201133056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317364733862912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62235816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8515495404544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377127289812480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370927170763776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325880275664384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365096731442176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 % _timestamp‚1.7884403498534122e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6200119048704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚2789.898271578 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚371283999822509 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62235816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6200119048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚370927144372224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54978535 _step‚271 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54978054 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49070795 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370927144372224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49070801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23170381 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6187268380333 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370927170763776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317364705728512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365096756870144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377127289812480 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8515541801472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚371286623535141 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23169903 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62215783 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6189866664997 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62215783 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377124663212032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54978535 _step‚271 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54978054 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 YJDeviceStatsMonitor.oSÛd""n_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49070795 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370927144372224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49070801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23170381 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6187268380333 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚370927170763776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317364705728512 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365096756870144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377127289812480 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8515541801472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚371286623535141 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23169903 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62215783 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6189866664997 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62215783 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377124663212032 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49070801 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377124663212032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49070795 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325880201133056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317364733862912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62235816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8515495404544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377127289812480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370927170763776 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325880275664384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365096731442176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 % _timestamp‚1.7884403498534122e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6200119048704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚2789.898271578 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚371283999822509 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0~c0]’—‚(:‡ ÞåÔà™œÉ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚97.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²-²˜‚(:‘ ÞåÔƒß…Ë network.recv‚ 1.027534e+06 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1870.31640625% disk./.usageGB‚1105.3496131896973 disk.sda3.in‚0' disk.nvme0c0n1.out‚26758.4404296875 disk.dm-1.out‚ 114.421875 disk.rbd3.in‚0 disk.nvme0n1.in‚1724.625! proc.memory.rssMB‚ 2753.609375 proc.cpu.threads‚17! disk.rbd2.out‚153.11962890625 cpu‚25.255399220334173) disk./.usagePercent‚74.18093999126052 disk.nvme0c0n1.in‚1724.625 disk.drbd1777.in‚0 disk.sda.out‚ 114.421875 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.806697845458984 network.sent‚ 7.188854e+06 disk.rbd2.in‚0% disk.nvme0n1.out‚26758.4404296875$ memory_percent‚88.58449459075928 disk.sda3.out‚ 114.421875 disk.rbd0.in‚0 disk.dm-0.in‚1724.625" disk.dm-0.out‚26758.4404296875 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.2265625Ä ,“2™- _wandbruntime‚2790  _runtime‚2790ùwYÍ'š‚N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54994046 _runtime‚2790.634704977 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377237061859840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365203072616448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111319048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31817350 trainer/global_step‚3349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377237061859840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚371035084077568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371035110469120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325975161695744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6199351168512 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317457093782016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49085031 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62234017 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54994481*0é­ '›‡N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49085031 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62234017 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54994481 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54994046 _runtime‚2790.634704977 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377237061859840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365203072616448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111319048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31817350 trainer/global_step‚3349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377237061859840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚371035084077568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371035110469120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325975161695744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6199351168512 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317457093782016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573X’‡L2œ- _wandbruntime‚2790  _runtime‚2790„]`VÐ _runtime‚2790.635690424 /loss/FRt2/detection_loss‚3.1016302545865377 /loss/FBt3/detection_loss‚2.6733012199401855 _step‚273 .loss/FRt1/detection_loss‚1.593396412248307 2loss/SRqqt4/assignment_loss‚1.7016751908374077 1loss/FBt1/assignment_loss‚0.23877217014687688 1loss/SRqqt1/detection_loss‚1.1816549647712347 2loss/SRqqt2/assignment_loss‚1.9404031249622886 1loss/FBt3/assignment_loss‚0.11747051955067686 0loss/SRqqt3/detection_loss‚2.389978716450353 0loss/SRqqt4/detection_loss‚1.982763985590059 .loss/FBt2/detection_loss‚3.084101974964142 % _timestamp‚1.7884403505911524e+09 0loss/FRt1/assignment_loss‚1.8097305437351794 2loss/SRqqt1/assignment_loss‚1.0512243247393407 1loss/FBt2/assignment_loss‚0.10464320012833923 0loss/FBt4/assignment_loss‚0.3592068023979664 .loss/FBt4/detection_loss‚3.617016887664795 trainer/global_step‚3349 -loss/FRt3/assignment_loss‚4.5876049552645 .loss/FRt3/detection_loss‚2.790065394129072 0loss/SRqqt2/detection_loss‚1.955062020283479 epoch‚2 0loss/FRt4/assignment_loss‚3.4517744546351223 1loss/SRqqt3/assignment_loss‚1.734594821449249 /loss/FBt1/detection_loss‚0.6454529820559883 %loss/total_loss‚48.03319519935154 0loss/FRt2/assignment_loss‚3.5462463816006977 .loss/FRt4/detection_loss‚2.375423897219741°Â"'"žÕ 2loss/SRqqt1/assignment_loss‚1.0512243247393407 1loss/FBt2/assignment_loss‚0.10464320012833923 0loss/FBt4/assignment_loss‚0.3592068023979664 .loss/FBt4/detection_loss‚3.617016887664795 trainer/global_step‚3349 -loss/FRt3/assignment_loss‚4.5876049552645 .loss/FRt3/detection_loss‚2.790065394129072 0loss/SRqqt2/detection_loss‚1.955062020283479 epoch‚2 0loss/FRt4/assignment_loss‚3.4517744546351223 1loss/SRqqt3/assignment_loss‚1.734594821449249 /loss/FBt1/detection_loss‚0.6454529820559883 %loss/total_loss‚48.03319519935154 0loss/FRt2/assignment_loss‚3.5462463816006977 .loss/FRt4/detection_loss‚2.375423897219741 _runtime‚2790.635690424 /loss/FRt2/detection_loss‚3.1016302545865377 /loss/FBt3/detection_loss‚2.6733012199401855 _step‚273 .loss/FRt1/detection_loss‚1.593396412248307 2loss/SRqqt4/assignment_loss‚1.7016751908374077 1loss/FBt1/assignment_loss‚0.23877217014687688 1loss/SRqqt1/detection_loss‚1.1816549647712347 2loss/SRqqt2/assignment_loss‚¿¦%91.9404031249622886 1loss/FBt3/assignment_loss‚0.11747051955067686 0loss/SRqqt3/detection_loss‚2.389978716450353 0loss/SRqqt4/detection_loss‚1.982763985590059 .loss/FBt2/detection_loss‚3.084101974964142 % _timestamp‚1.7884403505911524e+09 0loss/FRt1/assignment_loss‚1.8097305437351794‘ì;“Ÿ‚(:ˆ ¬ÞåÔãš‹É gpu.0.gpu‚59.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙòO! ‚(:– ¬ÞåÔ†­©Ë disk.rbd1.out‚ 1.2265625 disk.sda.out‚ 114.82421875 disk.drbd1777.in‚0! disk.rbd2.out‚154.30712890625 network.sent‚ 7.28587e+06 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚1724.625' disk.nvme0c0n1.out‚26758.7646484375 disk.rbd2.in‚0 disk.sda.in‚0$ memory_percent‚88.58628273010254# proc.memory.rssMB‚ 2753.61328125* proc.memory.percent‚16.806721687316895 disk.dm-1.out‚ 114.82421875) disk./.usagePercent‚74.18094383137354 disk.rbd0.in‚0 cpu‚25.25421778961423 disk.dm-0.in‚1724.625" disk.dm-0.out‚26758.7646484375 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 114.82421875( proc.memory.availableMB‚ 1870.0234375 proc.cpu.threads‚17 disk.sda2.in‚0% disk.nvme0n1.out‚26758.7646484375 disk.sda3.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1105.3496704101562 disk.sda1.in‚0 disk.rbd3.in‚0 network.recv‚ 1.033814e+06 disk.sda1.out‚0 disk.dm-1.in‚0\šyÏ“¡‚(:ˆ »ÞåÔâ÷ÈÉ gpu.0.gpu‚59.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚206.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.3638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02§f…!¢‚(:– »ÞåÔî·¹Ë% disk./.usageGB‚1105.3576431274414 disk.drbd1777.in‚0! disk.rbd2.out‚155.38134765625 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 115.41015625 cpu‚25.25305163286714 disk.sda.out‚ 115.41015625 network.sent‚ 7.384247e+06 network.recv‚ 1.040277e+06 disk.rbd2.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚26759.2998046875# memory_percent‚88.5850191116333 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚74.1814788871201 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚1724.625* proc.memory.percent‚16.806721687316895 disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625 disk.dm-1.out‚ 115.41015625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0" disk.dm-0.out‚26759.2998046875 disk.sda1.out‚0) proc.memory.availableMB‚ 1870.23046875# proc.memory.rssMB‚ 2753.61328125 disk.sda.in‚0' disk.nvme0c0n1.out‚26759.2998046875n2£- _wandbruntime‚2829  _runtime‚2829ãݳÖǤÁ % _timestamp‚1.7884403889852672e+09 _runtime‚2829.029941275 _step‚274 & lr-AdamW/pg1‚0.0003500547772230888 & lr-AdamW/pg2‚0.0003500547772230888 trainer/global_step‚3399ƒ3OHÌ¥Æ & lr-AdamW/pg2‚0.0003500547772230888 trainer/global_step‚3399 % _timestamp‚1.7884403889852672e+09 _runtime‚2829.029941275 _step‚274 & lr-AdamW/pg1‚0.0003500547772230888’”®±2¦- _wandbruntime‚2829  _runtime‚2829¹†âaþ'§øO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6281347389605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8641581265920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 trainer/global_step‚3399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382613279328256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63150868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32267954 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32267951 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63150868 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376324156030976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382613279328256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49782595 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370413790155776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6289149688832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49782601 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884403889862177e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63130835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330625717804032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370413815583744 _step‚275 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376692539260717 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376324129639424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23511243 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376695162973349 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382615905928704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49782601 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376324156030976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23510764 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6291749897728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6278749104941 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 _runtime‚2829.031213862 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55778715 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6291749897728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647p¡µI(¨ýO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376324156030976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23510764 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6291749897728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6278749104941 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 _runtime‚2829.031213862 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55778715 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6291749897728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6281347389605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8641581265920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 trainer/global_step‚3399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382613279328256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63150868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32267954 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32267951 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63150868 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376324156030976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382613279328256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49782595 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370413790155776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6289149688832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49782601 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884403889862177e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63130835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330625717804032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370413815583744 _step‚275 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376692539260717 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376324129639424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23511243 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376695162973349 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382615905928704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49782601 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074“q1# 2©- _wandbruntime‚2829  _runtime‚2829î†ÔèÚ ªN UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6290982017536 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚330720609758720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49796837 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376432095736320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376432069344768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370520156758016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277150 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6283176732481 _runtime‚2829.770423662 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23517476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322076524591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370520131330048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63149069 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6293582239744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322076496457216 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382725677976064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚276 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63169128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49796831 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6290982017536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63169128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376432069344768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55794623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49796837 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6280578447609 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376800709777657 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6293582239744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573M^ê‡ '«†N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382723051362304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330720535240704 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6280578447609 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376800709777657 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6293582239744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6290982017536 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚330720609758720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49796837 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74518016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376432095736320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376432069344768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370520156758016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277150 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6283176732481 _runtime‚2829.770423662 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23517476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322076524591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370520131330048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63149069 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6293582239744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322076496457216 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382725677976064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚276 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63169128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49796831 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6290982017536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63169128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376432069344768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55794623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49796837 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8767007704064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326603539213312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388104522044928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚492 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56579418 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382101078698925 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50494401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32727751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381721141298176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 _step‚279 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388104522044928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388101895444480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381721141298176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6380780537856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8767058295296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64045887 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326603511078912ìœ}DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8767007704064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326603539213312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388104522044928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚492 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56579418 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382101078698925 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50494401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32727751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381721141298176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 _step‚279 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388104522044928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388101895444480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381721141298176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6380780537856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8767058295296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64045887 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326603511078912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2868.051051152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64065920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚489 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6372828114213 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335370518782976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64065920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6380780537856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23852156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23851667 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚375730848869376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6383380746752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6383380746752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388101895444480 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540282 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚375730874297344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114560321 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50508637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32736950 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326695899132416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64084180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6374657457089 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56595341 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382209249215865 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64064121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335465484613120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335465405900800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388214294092288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381829081003520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381829054611968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64084180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8769557346304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388211667478528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884404289713142e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382211872928705 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8769506768384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6385213088768 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚375837215471616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6382612866560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50508637 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388214294092288 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326695927266816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6382612866560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ïíp '½‡N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335465484613120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335465405900800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388214294092288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381829081003520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚381829054611968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64084180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8769557346304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388211667478528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884404289713142e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚382211872928705 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8769506768384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6385213088768 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚375837215471616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6382612866560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50508637 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388214294092288 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326695927266816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6382612866560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64064121 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114592817 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50508631 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50508637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32736950 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326695899132416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64084180 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6374657457089 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56595341 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382209249215865 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64064121˜Hícþ2¾- _wandbruntime‚2869  _runtime‚2869ÒÚê-V¿Ð /loss/FRt2/detection_loss‚2.4571005689061205 /loss/FBt2/detection_loss‚3.0874715911017523 epoch‚2 trainer/global_step‚3449 2loss/SRqqt3/assignment_loss‚2.4193684564482782 2loss/SRqqt4/assignment_loss‚1.3367092423141003 .loss/FRt4/detection_loss‚2.431893886998296 _runtime‚2869.017699238 -loss/FRt3/assignment_loss‚4.4814885324902 0loss/FRt1/assignment_loss‚1.4261060789519666 2loss/SRqqt1/assignment_loss‚0.9434676696405266 0loss/FBt1/assignment_loss‚0.2141134477487689 0loss/SRqqt2/detection_loss‚2.067830416620994 1loss/SRqqt3/detection_loss‚2.7741128302389577 /loss/FBt3/detection_loss‚2.7903701066970825 0loss/FBt3/assignment_loss‚0.4806321619544178 0loss/FBt4/assignment_loss‚0.5616604029718373 .loss/FRt3/detection_loss‚2.446206616030799 /loss/FBt1/detection_loss‚0.6208380915955571 % _timestamp‚1.7884404289731185e+09 _step‚281 /loss/FBt2/assignment_loss‚0.288590504684382 %loss/total_loss‚48.00879566275869 /loss/FRt4/assignment_loss‚4.102780344585578 2loss/SRqqt2/assignment_loss‚1.6950927030447187 /loss/FRt1/detection_loss‚1.7108563497370364 1loss/SRqqt4/detection_loss‚2.1438779518008233 .loss/FBt4/detection_loss‚2.931602312458886 0loss/FRt2/assignment_loss‚3.4573472751223524 1loss/SRqqt1/detection_loss‚1.1392781206152656ìÛÒØ[ÀÕ -loss/FRt3/assignment_loss‚4.4814885324902 0loss/FRt1/assignment_loss‚1.4261060789519666 2loss/SRqqt1/assignment_loss‚0.9434676696405266 0loss/FBt1/assignment_loss‚0.2141134477487689 0loss/SRqqt2/detection_loss‚2.067830416620994 1loss/SRqqt3/detection_loss‚2.7741128302389577 /loss/FBt3/detection_loss‚2.7903701066970825 0loss/FBt3/assignment_loss‚0.4806321619544178 0loss/FBt4/assignment_loss‚0.5616604029718373 .loss/FRt3/detection_loss‚2.446206616030799 /loss/FBt1/detection_loss‚0.6208380915955571 % _timestamp‚1.7884404289731185e+09 _step‚281 /loss/FBt2/assignment_loss‚0.288590504684382 %loss/total_loss‚48.00879566275869 /loss/FRt4/assignment_loss‚4.102780344585578 2loss/SRqqt2/assignment_loss‚1.6950927030447187 /loss/FRt1/detection_loss‚1.7108563497370364 1loss/SRqqt4/detection_loss‚2.1438779518008233 .loss/FBt4/detection_loss‚2.931602312458886 0loss/FRt2/assignment_loss‚3.4573472751223524 1loss/SRqqt1/detection_loss‚1.1392781206152656 /loss/FRt2/detection_loss‚2.4571005689061205 /loss/FBt2/detection_loss‚3.0874715911017523 epoch‚2 trainer/global_step‚3449 2loss/SRqqt3/assignment_loss‚2.4193684564482782 2loss/SRqqt4/assignment_loss‚1.3367092423141003 .loss/FRt4/detection_loss‚2.431893886998296 _runtime‚2869.017699238™¡¦4Ç“Á‚(:ˆ ÷ÞåÔÅÓ É gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚263.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hmÝ‚(:‘ ÷ÞåÔÔüÂË% disk./.usageGB‚1105.3576393127441 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 117.5234375 network.recv‚ 1.059326e+06) disk./.usagePercent‚74.18147863111257 disk.rbd2.out‚159.7841796875 disk.sda1.out‚0% disk.nvme0n1.out‚26769.6552734375" disk.dm-0.out‚26769.6552734375 disk.rbd1.in‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚26769.6552734375 cpu‚25.24885918291953 disk.dm-1.out‚ 117.5234375 disk.drbd1777.out‚0 disk.rbd0.in‚0$ memory_percent‚88.58861923217773 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.806745529174805 disk.sda2.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625" proc.memory.rssMB‚ 2753.6171875 proc.cpu.threads‚17 network.sent‚ 7.510089e+06 disk.dm-0.in‚1724.625 disk.dm-1.in‚0 disk.sda.out‚ 117.5234375 disk.nvme0c0n1.in‚1724.625' proc.memory.availableMB‚ 1869.640625 disk.rbd1.out‚ 1.2265625 disk.rbd3.in‚0î× ‘Â(:† †ßåÔ¢ëÇÉ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â,ø²Ä‚(:’ †ßåÔ祸Ë disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚26769.99853515625 disk.sda3.out‚ 117.859375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26769.99853515625 disk.dm-0.in‚1724.625# disk.dm-0.out‚26769.99853515625 proc.cpu.threads‚17# memory_percent‚88.5871171951294 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚1724.625 disk.rbd2.in‚0 disk.dm-1.out‚ 117.859375) proc.memory.availableMB‚ 1869.88671875 network.sent‚ 7.61788e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚1724.625 cpu‚25.24775425644772) disk./.usagePercent‚74.18147863111257 disk.drbd1777.out‚0! disk.rbd2.out‚160.97607421875 disk.sda.out‚ 117.859375 disk.sda3.in‚0" proc.memory.rssMB‚ 2753.6171875 disk.rbd3.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.8067455Ðlk29174805 network.recv‚ 1.065214e+06% disk./.usageGB‚1105.3576393127441 disk.rbd1.out‚ 1.2265625$ 2Å- _wandbruntime‚2907  _runtime‚2907Ý1qÈÆ % _timestamp‚1.7884404671198866e+09 _runtime‚ 2907.16470101 _step‚282 ' lr-AdamW/pg1‚0.00034668389649607776 ' lr-AdamW/pg2‚0.00034668389649607776 trainer/global_step‚34997“óÍÇÇ ' lr-AdamW/pg1‚0.00034668389649607776 ' lr-AdamW/pg2‚0.00034668389649607776 trainer/global_step‚3499 % _timestamp‚1.7884404671198866e+09 _runtime‚ 2907.16470101 _step‚282š†ÐA^2È- _wandbruntime‚2907  _runtime‚2907”ž¦ý'É÷O RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331222941888512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64980972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6475011595776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.788440467120655e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _runtime‚2907.165575197 _step‚283 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51206195 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340115434035712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚393593138161152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387118126565376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24192432 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8892520281600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381047907582976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚393590511560704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6464308838821 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340115508567040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33187551 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6461710554157 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6475011595776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚393593138161152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331222913754112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51206201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24192892 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387509618137133 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51206201 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64980972 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387118126565376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64960939 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33187554 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387512241849765 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6472411386880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387118100173824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57380446 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381047933010944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚393590511560704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6472411386880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8892566678528 trainer/global_step‚3499ƒD¸Ú(ÊüO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331222941888512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64980972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6475011595776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.788440467120655e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _runtime‚2907.165575197 _step‚283 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51206195 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340115434035712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚393593138161152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387118126565376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24192432 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8892520281600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381047907582976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚393590511560704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6464308838821 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340115508567040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33187551 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6461710554157 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6475011595776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚393593138161152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331222913754112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51206201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24192892 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74531328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387509618137133 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51206201 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64980972 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387118126565376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64960939 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33187554 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387512241849765 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6472411386880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387118100173824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57380446 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712632 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚381047933010944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚393590511560704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6472411386880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8892566678528 trainer/global_step‚3499›ò§Ø2Ë- _wandbruntime‚2907  _runtime‚2907>Þ'Ì‚N REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51220437 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381154248757248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33196747 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24199791 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚393702910208512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚2907.902969857 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51220431 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6476843937792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6474243715584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116199604 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331315329942016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381154274185216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64999232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3499 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51220431 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57396541 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51220437 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚284 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24199340 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57396087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6476843937792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387226039879168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340210332008960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331315301807616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8895030201344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64979173 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64999232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116199604 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116219669 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331315329942016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381154274185216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64999232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3499 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51220431 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57396541 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segmentMXñð .large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51220437 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚284 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24199340 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57396087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6476843937792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387226039879168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340210332008960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331315301807616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8895030201344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64979173 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64999232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FsË2Î- _wandbruntime‚2907  _runtime‚2907 .'cÏÝ _step‚285 0loss/FRt2/assignment_loss‚3.1586617678403854 /loss/FRt3/detection_loss‚2.5683084653645025 1loss/FBt3/assignment_loss‚0.21314911532681435 1loss/FBt4/assignment_loss‚0.23155808572967848 1loss/SRqqt1/detection_loss‚0.9891394687195619 epoch‚2 % _timestamp‚1.7884404678601449e+09 2loss/SRqqt1/assignment_loss‚0.9549098997376859 1loss/SRqqt4/detection_loss‚1.9559592520290952 2loss/SRqqt2/assignment_loss‚1.5603847989382653 1loss/FBt1/assignment_loss‚0.10864421501951299 3loss/FBt2/assignment_loss‚0.0026702720555476845 /loss/FRt1/detection_loss‚1.6831721713946712 /loss/FRt2/detection_loss‚2.4005362594669517 /loss/FBt1/detection_loss‚0.5552866346449029 _runtime‚2907.904755512 0loss/FRt3/assignment_loss‚3.5513147028481087 /loss/FRt4/detection_loss‚2.9276449887288942 %loss/total_loss‚43.84382496614414 0loss/FRt1/assignment_loss‚1.3458688980148685 /loss/FBt2/detection_loss‚3.1359357297420503 /loss/FBt4/detection_loss‚2.6158795158068338 0loss/FRt4/assignment_loss‚3.5184830443726645 2loss/SRqqt3/assignment_loss‚1.8616232165405828 0loss/SRqqt3/detection_loss‚2.444103983140761 /loss/FBt3/detection_loss‚2.4529918084541955 trainer/global_step‚3499 2loss/SRqqt4/assignment_loss‚1.5678439084089026 0loss/SRqqt2/detection_loss‚2.039754763818704MþdöhÐâ 0loss/FRt1/assignment_loss‚1.3458688980148685 /loss/FBt2/detection_loss‚3.1359357297420503 /loss/FBt4/detection_loss‚2.6158795158068338 0loss/FRt4/assignment_loss‚3.5184830443726645 2loss/SRqqt3/assignment_loss‚1.8616232165405828 0loss/SRqqt3/detection_loss‚2.444103983140761 /loss/FBt3/detection_loss‚2.4529918084541955 trainer/global_step‚3499 2loss/SRqqt4/assignment_loss‚1.5678439084089026 0loss/SRqqt2/detection_loss‚2.039754763818704 _step‚285 0loss/FRt2/assignment_loss‚3.1586617678403854 /loss/FRt3/detection_loss‚2.5683084653645025 1loss/FBt3/assignment_loss‚0.21314911532681435 1loss/FBt4/assignment_loss‚0.23155808572967848 1loss/SRqqt1/detection_loss‚0.9891394687195619 epoch‚2 % _timestamp‚1.7884404678601449e+09 2loss/SRqqt1/assignment_loss‚0.9549098997376859 1loss/SRqqt4/detection_loss‚1.9559592520290952 2loss/SRqqt2/assignment_loss‚1.5603847989382653 1loss/FBt1/assignment_loss‚0.10864421501951299 3loss/FBt2/assignment_loss‚0.0026702720555476845 /loss/FRt1/detection_loss‚1.6831721713946712 /loss/FRt2/detection_loss‚2.4005362594669517 /loss/FBt1/detection_loss‚0.5552866346449029 _runtime‚2907.904755512 0loss/FRt3/assignment_loss‚3.5513147028481087 /loss/FRt4/detection_loss‚2.9276449887288942 %loss/total_loss‚43.84382496614414a\À“Ñ‚(:ˆ •ßåÔô®ÿÈ gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¥™ÄÒ‚(:‰ •ßåÔ‹à×Ë disk.sda1.in‚0* proc.memory.percent‚16.806745529174805$ disk./.usageGB‚1105.482578277588 disk.rbd2.in‚0% disk.nvme0c0n1.out‚26867.92578125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 118.4296875 disk.sda1.out‚0$ memory_percent‚88.58883380889893) proc.memory.availableMB‚ 1869.60546875' disk./.usagePercent‚74.189863389874 disk.sda3.out‚ 118.4296875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚1724.625 disk.rbd1.out‚1.3125 disk.sda.in‚0 network.sent‚ 7.623779e+06 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625# disk.nvme0n1.out‚26867.92578125 network.recv‚ 1.069196e+06 disk.dm-0.in‚1724.625 disk.dm-0.out‚26867.92578125 disk.dm-1.in‚0 disk.rbd1.in‚0! disk.rbd2.out‚161.98779296875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 118.4296875" proc.memory.rssMB‚ 2753.6171875 cpu‚25.246740111120424 proc.cpu.threads‚17•Ïö‰“Ó‚(:ˆ ¤ßåÔÜö€Ë gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚255.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r(³Ô‚(:ˆ ¤ßåÔ³·ÖÌ disk.dm-1.in‚0$ memory_percent‚88.58728408813477 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚ 7.732351e+06) disk./.usagePercent‚74.18984521333907! disk.rbd2.out‚164.18701171875 disk.sda.out‚ 118.80078125 disk.sda2.out‚0' proc.memory.availableMB‚ 1869.859375 cpu‚25.245637727661723 network.recv‚ 1.076077e+06$ disk./.usageGB‚1105.482307434082 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 26868.2265625* proc.memory.percent‚16.806745529174805 disk.rbd3.in‚0 disk.nvme0n1.in‚1724.625 disk.rbd0.out‚0 disk.rbd1.out‚1.375 disk.nvme0c0n1.in‚1724.625 disk.dm-0.in‚1724.625" disk.nvme0n1.out‚ 26868.2265625 disk.sda3.in‚0 disk.sda3.out‚ 118.80078125 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 26868.2265625 disk.dm-1.out‚ 118.80078125" proc.memory.rssMB‚ 2753.6171875 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0y)]+Õ‚(:’ ³ßåÔô¼„É gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚261.068' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.21359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çuâ\Ö‚(:“ ³ßåÔ ™‚Ë disk.sda2.out‚0" proc.memory.rssMB‚ 2753.6171875 disk.nvme0c0n1.in‚1724.625 disk.rbd1.out‚1.375 disk.sda.in‚0 disk.sda2.in‚0 network.recv‚ 1.08073e+06% disk./.usageGB‚1105.4824295043945* proc.memory.percent‚16.806745529174805) disk./.usagePercent‚74.18985340558017' disk.nvme0c0n1.out‚26868.7607421875 disk.nvme0n1.in‚1724.625% disk.nvme0n1.out‚26868.7607421875 disk.rbd0.in‚0 disk.sda3.in‚0 network.sent‚ 7.738644e+06 disk.dm-0.in‚1724.625" disk.dm-0.out‚26868.7607421875 disk.dm-1.out‚ 119.34765625$ memory_percent‚88.58582973480225 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1870.09765625 cpu‚25.244678153008493 disk.drbd1777.out‚0 disk.sda.out‚ 119.34765625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 119.34765625 disk.rbd1.in‚0 disk.drbd1777.in‚0! disk.rbd2.out‚165.19873046875qj†2×- _wandbruntime‚2946  _runtime‚2946wTbÇØÁ & lr-AdamW/pg2‚0.0003449504742427482 trainer/global_step‚3549 % _timestamp‚1.7884405062152421e+09 _runtime‚2946.259885024 _step‚286 & lr-AdamW/pg1‚0.0003449504742427482h˜ÒMÌÙÆ & lr-AdamW/pg1‚0.0003449504742427482 & lr-AdamW/pg2‚0.0003449504742427482 trainer/global_step‚3549 % _timestamp‚1.7884405062152421e+09 _runtime‚2946.259885024 _step‚286žóžÊg2Ú- _wandbruntime‚2946  _runtime‚2946;Â\ˆý'Û÷O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399081754277376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344860687562752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33647354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9018419627520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386364966296576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33647351 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392515085441024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399081754277376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65896024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24532792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65875991 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392515111832576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6566642444800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51917995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6564042235904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6564042235904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399079127676928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9018371133440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51917995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6566642444800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117814025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392918157575341 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6553191278765 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51918001 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24533255 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386364991724544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 $ _timestamp‚1.788440506215941e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58180143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51918001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392920781287973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6555789563429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _step‚287 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65896024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399079127676928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344860764191232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65875991 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0øcп§ÜüO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51918001 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24533255 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386364991724544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 $ _timestamp‚1.788440506215941e+09 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58180143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51918001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392920781287973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6555789563429 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _step‚287 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65896024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399079127676928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344860764191232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65875991 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399081754277376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76628480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344860687562752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33647354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9018419627520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386364966296576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33647351 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392515085441024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399081754277376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65896024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24532792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2036èæ[9637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65875991 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392515111832576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6566642444800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51917995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6564042235904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6564042235904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399079127676928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9018371133440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51917995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6566642444800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117814025 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392918157575341 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6553191278765 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039Ÿ_ }2Ý- _wandbruntime‚2946  _runtime‚2946_dÀÔ'Þ‚N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65914284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6568474786816 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9020877541888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335934732617216 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51932231 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392623025146368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399188899710976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392623051537920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58196588 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386471307470848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117846521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344955582024704 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393028951805121 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6557618906305 _runtime‚2946.997789959 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58196134 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65894225 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6565874564608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9020926022656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33656547 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24540038 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ì&¨¯ '߇N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344955582024704 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393028951805121 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6557618906305 _runtime‚2946.997789959 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58196134 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65894225 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6565874564608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9020926022656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117826456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33656547 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24540038 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65914284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6568474786816 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9020877541888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335934732617216 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51932231 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392623025146368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399188899710976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392623051537920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20059 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58196588 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386471307470848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117846521 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 LÉ]q2à- _wandbruntime‚2946  _runtime‚2946§8Åþ]á× /loss/FRt4/assignment_loss‚3.281486181418101 2loss/SRqqt2/assignment_loss‚1.8139905530517384 2loss/SRqqt4/assignment_loss‚1.6777435345782175 1loss/SRqqt4/detection_loss‚1.9502744634946187 _runtime‚2946.998869593 _step‚289 /loss/FBt2/detection_loss‚2.9009908702638416 epoch‚2 0loss/FRt1/assignment_loss‚1.5100168206474998 0loss/FRt2/assignment_loss‚3.5092876030177607 1loss/FBt2/assignment_loss‚0.42740717676093076 2loss/SRqqt1/assignment_loss‚0.8954780309890898 2loss/SRqqt3/assignment_loss‚2.0715753665057624 .loss/FRt3/detection_loss‚2.383598330693367 1loss/SRqqt2/detection_loss‚1.7149901251166553 .loss/FBt3/detection_loss‚2.752089038491249 1loss/FBt3/assignment_loss‚0.10917221760610119 0loss/FBt4/assignment_loss‚0.4160475589867149 /loss/FRt2/detection_loss‚2.4776840907771414 1loss/FBt1/assignment_loss‚0.13634495783315873 /loss/FRt4/detection_loss‚2.5214373615052965 /loss/FBt4/detection_loss‚2.2037744905267442 1loss/SRqqt1/detection_loss‚0.8432882692951422 /loss/FBt1/detection_loss‚0.5951304031255217 %loss/total_loss‚44.37581484827088 % _timestamp‚1.7884405069542904e+09 /loss/FRt3/assignment_loss‚3.986809036670587 1loss/SRqqt3/detection_loss‚2.4514456689357758 .loss/FRt1/detection_loss‚1.745752697979862 trainer/global_step‚3549Xûâ8bâÜ /loss/FRt4/assignment_loss‚3.281486181418101 2loss/SRqqt2/assignment_loss‚1.8139905530517384 2loss/SRqqt4/assignment_loss‚1.6777435345782175 1loss/SRqqt4/detection_loss‚1.9502744634946187 _runtime‚2946.998869593 _step‚289 /loss/FBt2/detection_loss‚2.9009908702638416 epoch‚2 0loss/FRt1/assignment_loss‚1.5100168206474998 0loss/FRt2/assignment_loss‚3.5092876030177607 1loss/FBt2/assignment_loss‚0.42740717676093076 2loss/SRqqt1/assignment_loss‚0.8954780309890898 2loss/SRqqt3/assignment_loss‚2.0715753665057624 .loss/FRt3/detection_loss‚2.383598330693367 1loss/SRqqt2/detection_loss‚1.7149901251166553 .loss/FBt3/detection_loss‚2.752089038491249 1loss/FBt3/assignment_loss‚0.10917221760610119 0loss/FBt4/assignment_loss‚0.4160475589867149 /loss/FRt2/detection_loss‚2.4776840907771414 1loss/FBt1/assignment_loss‚0.13634495783315873 /loss/FRt4/detection_loss‚2.5214373615052965 /loss/FBt4/detection_loss‚2.2037744905267442 1loss/SRqqt1/detection_loss‚0.8432882692951422 /loss/FBt1/detection_loss‚0.5951304031255217 %loss/total_loss‚44.37581484827088 % _timestamp‚1.7884405069542904e+09 /loss/FRt3/assignment_loss‚3.986809036670587 1loss/SRqqt3/detection_loss‚2.4514456689357758 .loss/FRt1/detection_loss‚1.745752697979862 trainer/global_step‚3549¡ï“Atã‚(:… ÂßåÔ…ŠÿÈ gpu.0.gpu‚2.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01þø±!ä‚(:– ÂßåÔüÏšË disk.sda2.out‚0 disk.sda3.out‚ 119.8359375 disk.dm-1.out‚ 119.8359375 network.sent‚ 7.743861e+06 disk.sda.out‚ 119.8359375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚1724.625 network.recv‚ 1.084325e+06) disk./.usagePercent‚74.19824226046214 disk.rbd1.out‚ 1.4072265625 disk.nvme0n1.in‚1724.625% disk.nvme0n1.out‚26966.2607421875 disk.rbd0.out‚0 disk.dm-0.in‚1724.625* proc.memory.percent‚16.806745529174805 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2753.6171875 cpu‚25.2436955146198! disk.rbd2.out‚166.39013671875 disk.sda1.in‚0' disk.nvme0c0n1.out‚26966.2607421875" disk.dm-0.out‚26966.2607421875 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1870.17578125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.drbd1777.in‚0$ memory_percent‚88.58535289764404% disk./.usageGB‚1105.6074295043945 disk.sda.in‚0 disk.sda1.out‚0‚ ¨ˆ“å‚(:ˆ ÑßåÔ£ªùÈ gpu.0.gpu‚89.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€z¬~æ‚(:’ ÑßåÔÐâÌÊ disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 120.34375 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2753.6171875& disk.nvme0n1.out‚26966.91845703125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚26966.91845703125 disk.sda2.in‚0* proc.memory.percent‚16.806745529174805 disk.nvme0c0n1.in‚1724.625 disk.dm-1.out‚ 120.34375$ memory_percent‚88.58623504638672 network.recv‚ 1.090793e+06% disk./.usageGB‚1105.6076736450195 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.4072265625! disk.rbd2.out‚167.40185546875& proc.memory.availableMB‚ 1870.03125# disk.dm-0.out‚26966.91845703125 disk.rbd2.in‚0 disk.sda.out‚ 120.34375 disk.sda2.out‚0 cpu‚25.242653233969328 network.sent‚ 7.852433e+06 disk.nvme0n1.in‚1724.625 disk.drbd1777.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.19825864494433 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0vÀ„žç‚(:“ àßåÔ߯ÚÈ gpu.0.gpu‚71.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚248.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.782399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êv5[è‚(:” àßåÔÅáùË disk.drbd1777.out‚0* proc.memory.percent‚16.806769371032715% disk./.usageGB‚1105.8586921691895 disk.sda2.out‚0 disk.nvme0n1.in‚1724.625 disk.dm-1.in‚0 disk.rbd1.out‚ 1.47021484375! disk.rbd2.out‚169.54248046875 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚1724.625 disk.sda.in‚0# proc.memory.rssMB‚ 2753.62109375 proc.cpu.threads‚17 disk.sda3.out‚ 120.84375 disk.rbd3.in‚0 disk.dm-1.out‚ 120.84375$ memory_percent‚88.58480453491211 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 120.84375 network.sent‚ 7.858204e+06 disk.rbd0.out‚0) disk./.usagePercent‚74.21510470871995 disk.sda1.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚26967.18017578125# disk.dm-0.out‚26967.18017578125 disk.rbd1.in‚0' proc.memory.availableMB‚ 1870.265625 cpu‚25.24167040018194 network.recv‚ 1.094818e+06& disk.nvme0n1.out‚26967.18017578125 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0ðÕüò2é- _wandbruntime‚2985  _runtime‚2985.„¤Èê trainer/global_step‚3599 $ _timestamp‚1.788440545104227e+09 _runtime‚2985.148875355 _step‚290 ' lr-AdamW/pg1‚0.00034318554350580083 ' lr-AdamW/pg2‚0.00034318554350580083›÷Ð'ÍëÇ $ _timestamp‚1.788440545104227e+09 _runtime‚2985.148875355 _step‚290 ' lr-AdamW/pg1‚0.00034318554350580083 ' lr-AdamW/pg2‚0.00034318554350580083 trainer/global_step‚3599¢lLW-2ì- _wandbruntime‚2985  _runtime‚29854ßåûØíøO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚404570370393600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119420838 R>D%‚›³&$eviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404567743793152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6658273293824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52629801 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884405451049185e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397912070708224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58980224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340461719104512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9144109043200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6655673084928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66791043 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6644672003373 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚291 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6655673084928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398329320726181 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397912070708224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404570370393600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24872597 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600208896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚404570370393600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600208896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48494080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119420838 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404567743793152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6658273293824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52629801 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884405451049185e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788340736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20033 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397912070708224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58980224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340461719104512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9144109043200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6655673084928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66791043 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6644672003373 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚291 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6655673084928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788340736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398329320726181 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397912070708224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569408£;Îg2ï- _wandbruntime‚2985  _runtime‚2985d¶M9'ð‚N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24879293 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349700676184064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349700752799232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398434867530489 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6660105635840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34116350 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚3599 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58995640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚468 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340554107158016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404677515827200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398020010413568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404680142440960 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6649099630913 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66809277 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340554135292416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884405458357232e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58996108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34116347 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391788391612416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚391788366184448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398020036805120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9146569026048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24879758 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398437491243329 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52644037 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52644031{"ôz ñ‡N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66809277 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340554135292416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884405458357232e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58996108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34116347 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391788391612416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚391788366184448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398020036805120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9146569026048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ¾l5x“_step‚292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24879758 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398437491243329 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626613760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52644037 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52644031 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569408 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626613760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24879293 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349700676184064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349700752799232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398434867530489 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20059 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6660105635840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34116350 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚3599 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119473373 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58995640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚468 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788340736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340554107158016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404677515827200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398020010413568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465229 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788340736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404680142440960 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76615168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6649099630913 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074¤ç=2ò- _wandbruntime‚2985  _runtime‚2985ÿT]ó× 2loss/SRqqt3/assignment_loss‚1.9196637876935907 /loss/FRt3/detection_loss‚2.8198663610400576 0loss/SRqqt1/detection_loss‚1.050144204923085 1loss/SRqqt4/detection_loss‚1.8320500876369148 % _timestamp‚1.7884405458375423e+09 1loss/SRqqt3/detection_loss‚2.6598602033950187 1loss/FBt3/assignment_loss‚0.07215401958383154 0loss/FRt2/assignment_loss‚3.2783394235631693 2loss/SRqqt2/assignment_loss‚1.1069478503029262 1loss/FBt2/assignment_loss‚0.46900341656458167 _step‚293 /loss/FBt1/detection_loss‚0.5712796550443662 %loss/total_loss‚44.04080013280027 0loss/FRt4/assignment_loss‚2.8808231721533106 /loss/FBt3/detection_loss‚2.5030829191207884 epoch‚2 trainer/global_step‚3599 0loss/FBt4/assignment_loss‚0.1859055773820728 .loss/FRt2/detection_loss‚2.367430242507354 -loss/FBt2/detection_loss‚2.50232309954507 /loss/FBt4/detection_loss‚3.8365493565797806 /loss/FRt1/detection_loss‚1.5200393438643338 0loss/FRt3/assignment_loss‚3.9323560866442593 1loss/FBt1/assignment_loss‚0.12544834638482003 .loss/FRt4/detection_loss‚2.395309862304241 0loss/FRt1/assignment_loss‚1.5146626392189337 _runtime‚2985.882102782 2loss/SRqqt1/assignment_loss‚1.1813652157251324 2loss/SRqqt4/assignment_loss‚1.1538313286668012 1loss/SRqqt2/detection_loss‚2.16236393295583263åXzbôÜ .loss/FRt4/detection_loss‚2.395309862304241 0loss/FRt1/assignment_loss‚1.5146626392189337 _runtime‚2985.882102782 2loss/SRqqt1/assignment_loss‚1.1813652157251324 2loss/SRqqt4/assignment_loss‚1.1538313286668012 1loss/SRqqt2/detection_loss‚2.1623639329558326 2loss/SRqqt3/assignment_loss‚1.9196637876935907 /loss/FRt3/detection_loss‚2.8198663610400576 0loss/SRqqt1/detection_loss‚1.050144204923085 1loss/SRqqt4/detection_loss‚1.8320500876369148 % _timestamp‚1.7884405458375423e+09 1loss/SRqqt3/detection_loss‚2.6598602033950187 1loss/FBt3/assignment_loss‚0.07215401958383154 0loss/FRt2/assignment_loss‚3.2783394235631693 2loss/SRqqt2/assignment_loss‚1.1069478503029262 1loss/FBt2/assignment_loss‚0.46900341656458167 _step‚293 /loss/FBt1/detection_loss‚0.5712796550443662 %loss/total_loss‚44.04080013280027 0loss/FRt4/assignment_loss‚2.8808231721533106 /loss/FBt3/detection_loss‚2.5030829191207884 epoch‚2 trainer/global_step‚3599 0loss/FBt4/assignment_loss‚0.1859055773820728 .loss/FRt2/detection_loss‚2.367430242507354 -loss/FBt2/detection_loss‚2.50232309954507 /loss/FBt4/detection_loss‚3.8365493565797806 /loss/FRt1/detection_loss‚1.5200393438643338 0loss/FRt3/assignment_loss‚3.9323560866442593 1loss/FBt1/assignment_loss‚0.12544834638482003¥ª9ï“õ‚(:ˆ ïßåÔ«§ŸÉ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚280.694' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'º<¤$ö‚(:™ ïßåÔÏì›Ì network.sent‚ 7.864407e+06 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.47021484375 disk.sda3.in‚0 proc.cpu.threads‚17! disk.rbd2.out‚170.55419921875 disk.sda.out‚ 121.5546875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚27064.9892578125 cpu‚25.240715913537812 network.recv‚ 1.099457e+06( disk./.usagePercent‚74.2150363547083 disk.sda.in‚0% disk.nvme0n1.out‚27064.9892578125 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚1724.625 disk.nvme0c0n1.in‚1724.625 disk.dm-1.out‚ 121.5546875* proc.memory.percent‚16.806769371032715% disk./.usageGB‚1105.8576736450195 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 121.5546875 disk.dm-0.in‚1724.625$ memory_percent‚88.58664035797119) proc.memory.availableMB‚ 1869.96484375 disk.rbd0.in‚0" disk.dm-0.out‚27064.9892578125# proc.memory.rssMB‚ 2753.62109375j±“‘÷‚(:† þßåÔÀØëÈ gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚39.0 gpu.0.powerWatts‚90.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åýÿø‚(: þßåÔ¤ÚøË disk.rbd3.out‚0! disk.rbd2.out‚171.74951171875 disk.sda2.out‚0 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 121.85546875 disk.dm-0.in‚1724.625 disk.sda.out‚ 121.85546875$ memory_percent‚88.88931274414062 proc.memory.rssMB‚ 2803.15625* proc.memory.percent‚17.109107971191406 cpu‚25.2522723359099" disk.dm-0.out‚27065.3486328125 disk.dm-1.out‚ 121.85546875 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 7.972173e+06% disk./.usageGB‚1105.8576736450195' disk.nvme0c0n1.out‚27065.3486328125 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚1724.625% disk.nvme0n1.out‚27065.3486328125$ proc.memory.availableMB‚1820.375 network.recv‚ 1.105163e+06 disk.rbd1.out‚ 1.533203125 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚74.2150363547083 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0ì'¤—’ù‚(:‡ àåÔù¶œÉ gpu.0.gpu‚91.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.933 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×bØú‚(:… àåÔЩ¬Ë disk.sda3.out‚ 122.421875) disk./.usagePercent‚74.21493599975489% disk./.usageGB‚1105.8561782836914 disk.drbd1777.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚173.77294921875 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚1724.625$ memory_percent‚88.83476257324219! proc.memory.rssMB‚ 2794.109375 proc.cpu.threads‚17 disk.rbd1.out‚ 1.533203125 disk.sda.out‚ 122.421875 disk.rbd3.out‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 27163.0859375 cpu‚25.27386190635694 network.sent‚ 7.978351e+06 network.recv‚ 1.109592e+06" disk.nvme0n1.out‚ 27163.0859375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 122.421875% proc.memory.availableMB‚ 1829.3125* proc.memory.percent‚17.053890228271484 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚1724.625 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0 disk.dm-0.out‚ 27163.0859375wÏAû‚(:… œàåÔ³ØðÈ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚36.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆŒwQü‚(:† œàåÔ¼ÙºÌ disk.drbd1777.in‚0! disk.rbd2.out‚174.90185546875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚1724.625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda1.out‚0) disk./.usagePercent‚74.21493599975489$ disk.nvme0c0n1.out‚ 27163.3828125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2794.109375 cpu‚25.2961880306408 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 122.796875$ memory_percent‚88.83280754089355 disk.nvme0n1.in‚1724.625 disk.rbd1.in‚0 network.recv‚ 1.11293e+06% disk./.usageGB‚1105.8561782836914" disk.nvme0n1.out‚ 27163.3828125 disk.dm-1.out‚ 122.796875 network.sent‚ 7.983159e+06 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.dm-0.out‚ 27163.3828125 disk.rbd1.out‚ 1.533203125( proc.memory.availableMB‚ 1829.6328125* proc.memory.percent‚17.053890228271484 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 122.796875B¬z2ý- _wandbruntime‚3057  _runtime‚3057˜$çšf*þàT TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012352 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121237802 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚398089633926144 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788340736 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 110782464 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23668457472 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚67675587 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24123 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚67675587 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚346085748104192 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6719725591805 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚404548579656704 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚34753210 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚355323811410944 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚404811331651409 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚53562215 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚53562210 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚67659614 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚34753212 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚398089657256960 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2647 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚404548555362304 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788340736 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9237952524288 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2647 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚53562210 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15973 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133460 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802644 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 $ _timestamp‚1.788440617641766e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569408 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚134 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3626 _runtime‚3057.686853703 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6730883529728 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2686470144 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚812 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2647 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2647`…ÅzÕÿåT OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚136 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚404809359517949 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6730883529728 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2686470144 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚812 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2647 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2647 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012352 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121237802 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚398089633926144 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121221824 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788340736 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 110782464 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23668457472 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚67675587 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24123 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚67675587 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚346085748104192 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6719725591805 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚404548579656704 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚34753210 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚355323811410944 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚404811331651409 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚53562215 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚53562210 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚67659614 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚34753212 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚398089657256960 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2647 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚404548555362304 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788340736 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9237952524288 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2647 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚53562210 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15973 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133460 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802644 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 $ _timestamp‚1.788440617641766e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569408 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚134 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3626 _runtime‚3057.686853703 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2647 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6722147080337 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1573 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6731357732864 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6720198277693 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 110782464 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15973 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788340736 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950717952 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚53573104 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133460 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3298820096 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚25264609 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚404622338244608 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 121239932 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9238426727424 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1573 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚134 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚346147730458624 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚411353671683072 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚60025885 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚136 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23668457472 _runtime‚3058.664692491 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚25264743 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚346147712809984 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121255910 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚404622313950208 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6733308450816 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1411 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚34761142 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚355386139537408 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2647 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630564 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950717952j¡ÙÒv)‚ðR ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9238426727424 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1573 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚134 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚346147730458624 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚411353671683072 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚60025885 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚136 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23668457472 _runtime‚3058.664692491 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚25264743 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚346147712809984 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121255910 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚404622313950208 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6733308450816 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1411 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚34761142 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚355386139537408 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2647 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630564 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569408 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950717952 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚404622338244608 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚67682801 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚411353671683072 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚67666828 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚411355646695424 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23668457472 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚67666828 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6731357732864 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9238537509888 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚53573109 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚404622313950208 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569408 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚34761140 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚67682801 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚411355646695424 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24123 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚53573109 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚355386267968512 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2686470144 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚53573104 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 6733308450816 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚60025749 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2647 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6722147080337 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1573 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6731357732864 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6720198277693 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 110782464 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15973 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788340736 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950717952 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚53573104 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133460 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3298820096 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚25264609 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚404622338244608 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 121239932§÷šÊö‘ƒ‚(:† «àåÔ—ÔŒÉ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¡î¡!„‚(:– «àåÔýÅË disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625' disk.nvme0c0n1.out‚27165.7763671875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda3.out‚ 123.4140625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.053890228271484 cpu‚25.316873792984712% disk.nvme0n1.out‚27165.7763671875 disk.dm-1.out‚ 123.4140625) proc.memory.availableMB‚ 1829.03515625 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2794.109375% disk./.usageGB‚1105.9920349121094 disk.sda.out‚ 123.4140625 disk.nvme0n1.in‚1724.625 disk.rbd0.out‚0" disk.dm-0.out‚27165.7763671875 disk.rbd2.in‚0! disk.rbd2.out‚176.92529296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚ 7.989347e+06) disk./.usagePercent‚74.22405345207932 disk.rbd3.in‚0 disk.dm-0.in‚1724.625 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.117511e+06$ memory_percent‚88.83645534515381 disk.drbd1777.out‚0 disk.sda.in‚0ÖG`2…- _wandbruntime‚3072  _runtime‚3072ü Í5†­§ A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018054697662591934 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5330460071563721 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29374024271965027 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9795918464660645 particle/accuracy_2_of_3‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2916666567325592 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17014026641845703 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03630952164530754 +jet/accuracy_2_of_5‚0.40922296047210693 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4807233512401581 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 _step‚296 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4008030295372009 6"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974365234375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48981478810310364 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4956825375556946 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18333332240581512 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9952513575553894 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22489021718502045 6"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4786376953125 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011520768515765667 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9800626039505005 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13010478019714355 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 (particle/accuracy‚0.8525338768959045 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.8703704476356506 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12440763413906097 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4092493951320648 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20729701220989227 +jet/accuracy_2_of_4‚0.35964709520339966 particle/accuracy_3_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.132716104388237 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9823784828186035 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 jet/accuracy_5_of_7‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.971129834651947 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.443452388048172 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21674378216266632 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 % _timestamp‚1.7884406320482585e+09 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18218326568603516 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4177631437778473 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt0FBŒÍ~t/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1734873354434967 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13571549952030182 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4956825375556946 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14000001549720764 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23714213073253632 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4707437753677368 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06737589091062546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08883127570152283 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0593353733420372 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5008927583694458 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.48316431045532227 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1904040277004242 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 +jet/accuracy_2_of_3‚0.21548064053058624 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25993800163269043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9899402260780334 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48258674144744873 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07399425655603409 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2831100821495056 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9823206663131714 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13391390442848206 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9608162641525269 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992345929145813 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9584485292434692 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.077998086810112 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6435739398002625 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018140587955713272 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28550711274147034 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.48514631390571594 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22756487131118774 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12226716428995132 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5020268559455872 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5241169929504395 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.994866669178009 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4241379499435425 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4939059317111969 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.866430401802063 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3333333134651184 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8895168900489807 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3799472153186798 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49702227115631104 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2532455027103424 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47075119614601135 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.403160959482193 *particle/sensitivity‚0.267483115196228 (particle/f_score‚0.36137792468070984 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1303761899471283 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1269773691892624 ,jet/accuracy_3_of_3‚0.020613383501768112 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1483435034751892 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40842410922050476 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4583664536476135 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46708542108535767 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3086419999599457 )loss/val_total_loss‚44.77353588007015 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.969994068145752 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.4223770499229431 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10555581748485565 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2597739100456238 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45831745862960815 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029169663786888123 -jet/accuracy_4_of_4‚0.0011337868636474013 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18321502208709717 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.42810457944869995 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _runtime‚3072.095630913 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4474446773529053 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48137497901916504 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4613913893699646 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.955459713935852 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015133246779441833 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4686926603317261 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994380533695221 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15372774004936218 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14040009677410126 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9722321033477783 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05948500707745552 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29878923296928406 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18122349679470062 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4800795614719391 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43954604864120483 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_6_of_7‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5009409189224243 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5250734686851501 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 epoch‚2 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4595959484577179 +jet/accuracy_3_of_4‚0.05219832435250282 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9823470115661621 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6729850172996521 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9519097208976746 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 jet/accuracy_3_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17166799306869507 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.39917707443237305 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4773891270160675 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.426361620426178 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13003000617027283 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4309513568878174 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1356711983680725 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9801034331321716 *jet/accuracy_1_of_1‚0.4728148579597473 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ,jet/accuracy_4_of_5‚0.008064515888690948 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5005704164505005 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13010478019714355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4790727496147156 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28409090638160706 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9806592464447021 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4099999964237213 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_1_of_7‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9419674277305603 particle/accuracy_3_of_6‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8703703880310059 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4960165023803711 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9945815801620483 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03116999939084053 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.42976880073547363 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18352888524532318 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19195400178432465 +jet/accuracy_2_of_2‚0.09900315850973129 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4166666865348816 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0836130678653717 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28611117601394653 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5054084062576294 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47889402508735657 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.804959774017334 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3262091279029846 particle/accuracy_5_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21331410109996796 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47601065039634705 4 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974365234375 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4415094256401062 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8552631735801697 particle/accuracy_2_of_6‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09003831446170807 jet/accuracy_4_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.091812863945961 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004342474974691868 jet/accuracy_5_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2405582219362259 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11234252154827118 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1375902146100998 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956610798835754 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4589197337627411 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07547779381275177 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5096856355667114 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34062501788139343 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026626482605934143 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0019841270986944437 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15237635374069214 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48666852712631226 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.075396828353405 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948969483375549 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21029195189476013 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8802083134651184 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06390256434679031 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.479166716337204 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4665186107158661 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_6‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3568401038646698 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32882729172706604 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4934616982936859 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1626453399658203 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2261904776096344 particle/accuracy_4_of_4‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15373311936855316 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 particle/accuracy_7_of_7‚0.5 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9903439283370972 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.46067526936531067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13644537329673767 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9969283938407898 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.8645833730697632 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4355725049972534 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012433863244950771 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.49844786524772644 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4840277433395386 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18265169858932495 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30401793122291565 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9806631803512573 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11072085797786713 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07818930596113205 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1369975358247757 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007220654282718897 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09248550981283188 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9755445122718811 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9810588955879211 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11522988975048065 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9920635223388672 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5385802388191223 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_7‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03188834711909294 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30570024251937866 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4892856776714325 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4171540439128876 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.37984493374824524 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9821040034294128 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8533223271369934 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 +jet/accuracy_3_of_6‚0.09615384787321091 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚3626 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22384995222091675 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9703785181045532 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_2‚0.7357597947120667 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 jet/accuracy_6_of_6‚0 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18047097325325012 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3243386447429657 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4881283938884735 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4519519805908203 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4710697829723358 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.982207715511322 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0Ÿ½1s‡²§ ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4474446773529053 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.481374979ŽwkMù01916504 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4613913893699646 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.955459713935852 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015133246779441833 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4686926603317261 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994380533695221 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15372774004936218 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14040009677410126 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9722321033477783 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05948500707745552 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29878923296928406 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18122349679470062 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4800795614719391 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43954604864120483 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_6_of_7‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5009409189224243 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5250734686851501 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 epoch‚2 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4595959484577179 +jet/accuracy_3_of_4‚0.05219832435250282 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9823470115661621 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6729850172996521 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9519097208976746 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 jet/accuracy_3_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17166799306869507 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.39917707443237305 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4773891270160675 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.426361620426178 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13003000617027283 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4309513568878174 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1356711983680725 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9801034331321716 *jet/accuracy_1_of_1‚0.4728148579597473 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ,jet/accuracy_4_of_5‚0.008064515888690948 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5005704164505005 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13010478019714355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4790727496147156 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28409090638160706 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9806592464447021 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4099999964237213 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_1_of_7‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9419674277305603 particle/accuracy_3_of_6‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8703703880310059 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4960165023803711 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9945815801620483 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03116999939084053 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.42976880073547363 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18352888524532318 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19195400178432465 +jet/accuracy_2_of_2‚0.09900315850973129 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4166666865348816 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0836130678653717 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28611117601394653 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5054084062576294 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47889402508735657 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.804959774017334 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3262091279029846 particle/accuracy_5_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21331410109996796 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47601065039634705 4 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974365234375 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4415094256401062 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8552631735801697 particle/accuracy_2_of_6‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09003831446170807 jet/accuracy_4_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.091812863945961 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004342474974691868 jet/accuracy_5_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2405582219362259 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11234252154827118 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1375902146100998 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956610798835754 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4589197337627411 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07547779381275177 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5096856355667114 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34062501788139343 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026626482605934143 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0019841270986944437 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15237635374069214 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48666852712631226 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.075396828353405 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948969483375549 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21029195189476013 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8802083134651184 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06390256434679031 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.479166716337204 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4665186107158661 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_6‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3568401038646698 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32882729172706604 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4934616982936859 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1626453399658203 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2261904776096344 particle/accuracy_4_of_4‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15373311936855316 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 particle/accuracy_7_of_7‚0.5 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9903439283370972 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.46067526936531067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13644537329673767 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9969283938407898 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.8645833730697632 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4355725049972534 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012433863244950771 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.49844786524772644 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4840277433395386 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18265169858932495 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30401793122291565 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9806631803512573 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11072085797786713 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07818930596113205 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1369975358247757 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007220654282718897 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09248550981283188 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9755445122718811 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9810588955879211 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11522988975048065 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9920635223388672 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5385802388191223 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_7‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03188834711909294 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30570024251937866 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4892856776714325 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4171540439128876 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.37984493374824524 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9821040034294128 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8533223271369934 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 +jet/accuracy_3_of_6‚0.09615384787321091 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚3626 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22384995222091675 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9703785181045532 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_2‚0.7357597947120667 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 jet/accuracy_6_of_6‚0 particle/accuracy_4_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703880310059 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18047097325325012 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3243386447429657 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4881283938884735 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4519519805908203 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4710697829723358 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.982207715511322 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018054697662591934 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5330460071563721 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.29374024271965027 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9795918464660645 particle/accuracy_2_of_3‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2916666567325592 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17014026641845703 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03630952164530754 +jet/accuracy_2_of_5‚0.40922296047210693 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4807233512401581 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 _step‚296 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4008030295372009 6"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974365234375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48981478810310364 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4956825375556946 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18333332240581512 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9952513575553894 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22489021718502045 6"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4786376953125 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011520768515765667 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9800626039505005 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13010478019714355 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 (particle/accuracy‚0.8525338768959045 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.8703704476356506 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12440763413906097 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4092493951320648 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20729701220989227 +jet/accuracy_2_of_4‚0.35964709520339966 particle/accuracy_3_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.132716104388237 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9823784828186035 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 jet/accuracy_5_of_7‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.971129834651947 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.443452388048172 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21674378216266632 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 % _timestamp‚1.7884406320482585e+09 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18218326568603516 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4177631437778473 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1734873354434967 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13571549952030182 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4956825375556946 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14000001549720764 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23714213073253632 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4707437753677368 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06737589091062546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08883127570152283 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0593353733420372 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5008927583694458 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.48316431045532227 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1904040277004242 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 +jet/accuracy_2_of_3‚0.21548064053058624 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25993800163269043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9899402260780334 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48258674144744873 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07399425655603409 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2831100821495056 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9823206663131714 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13391390442848206 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9608162641525269 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992345929145813 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9584485292434692 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.077998086810112 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6435739398002625 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018140587955713272 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28550711274147034 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.48514631390571594 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.22756487131118774 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ) ?ËÔMPurity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12226716428995132 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5020268559455872 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5241169929504395 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.994866669178009 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4241379499435425 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4939059317111969 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.866430401802063 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3333333134651184 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8895168900489807 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3799472153186798 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49702227115631104 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2532455027103424 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47075119614601135 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.403160959482193 *particle/sensitivity‚0.267483115196228 (particle/f_score‚0.36137792468070984 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1303761899471283 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1269773691892624 ,jet/accuracy_3_of_3‚0.020613383501768112 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1483435034751892 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40842410922050476 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4583664536476135 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46708542108535767 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3086419999599457 )loss/val_total_loss‚44.77353588007015 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.969994068145752 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.4223770499229431 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10555581748485565 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2597739100456238 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45831745862960815 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029169663786888123 -jet/accuracy_4_of_4‚0.0011337868636474013 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18321502208709717 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.42810457944869995 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _runtime‚3072.095630913¨”nMOšˆ‚(: ºàåÔ¤Ó—É gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚33.0 gpu.0.powerWatts‚77.29' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.458000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ò 0‰‚(:¥ ºàåÔéÒ³Ë' disk.nvme0c0n1.out‚27263.3232421875 disk.drbd1777.out‚0# memory_percent‚99.9760389328003 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.dm-0.in‚1724.625 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 3.92578125 cpu‚25.334445004403456 network.sent‚ 8.096835e+06 network.recv‚ 1.123344e+06% disk.nvme0n1.out‚27263.3232421875 disk.sda3.out‚ 124.58984375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625 disk.rbd2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚1724.625 disk.rbd1.in‚0" disk.dm-0.out‚27263.3232421875! disk.rbd2.out‚178.05029296875 disk.sda.in‚0.5625 disk.sda.out‚ 124.58984375 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.sda2.in‚0% disk./.usageGB‚1105.9910163879395# proc.memory.rssMB‚ 4662.27734375) disk./.usagePercent‚74.22398509806769 disk.rbd0.out‚0 disk.dm-1.out‚ 124.58984375 disk.rbd3.in‚0) proc.memory.percent‚28.45628261566162 §ú¤“Š‚(:ˆ ÉàåÔžß¨É gpu.0.gpu‚70.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚43.0 gpu.0.powerWatts‚225.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.0874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ѫ6‹‚(:« ÉàåÔñ·äÊ network.recv‚ 1.130006e+06 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 3747.03515625 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚1724.625 disk.dm-0.in‚1724.625 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625* proc.memory.percent‚17.053890228271484 cpu‚25.332685538550834 network.sent‚ 8.221062e+06% disk./.usageGB‚1107.8165588378906# disk.dm-0.out‚32734.62841796875 disk.dm-1.out‚ 125.04296875! proc.memory.rssMB‚ 2794.109375! disk.rbd2.out‚179.06201171875 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 125.04296875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚74.34649878361604 disk.sda.out‚ 125.04296875 disk.sda2.in‚0& disk.nvme0n1.out‚32734.62841796875 disk.rbd0.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚32734.62841796875$ memory_percent‚77.12991237640381 proc.cpu.threads‚178¦2Œ- _wandbruntime‚3099  _runtime‚3099x³ãÉà ' lr-AdamW/pg2‚0.00034138947209415586 trainer/global_step‚3649 % _timestamp‚1.7884406597759438e+09 _runtime‚3099.820637878 _step‚297 ' lr-AdamW/pg1‚0.00034138947209415586>vÙÎŽÈ ' lr-AdamW/pg1‚0.00034138947209415586 ' lr-AdamW/pg2‚0.00034138947209415586 trainer/global_step‚3649 % _timestamp‚1.7884406597759438e+09 _runtime‚3099.820637878 _step‚297©aOì2- _wandbruntime‚3099  _runtime‚3099îýøþ'øO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414733038632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54027923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407955869302784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358286997495808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _runtime‚3099.821571487 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6779795930624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54027923 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348986025667584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚401409153074176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414735665233408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777195721216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60530876 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414735665233408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68162512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚408175102120474 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68182546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35066537 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414733038632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408177725833110 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122210469 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70336512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚407955869302784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9300971828224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6765949046298 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚409 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348986053801984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407955842911232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60530464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25464339 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JË(‘ýO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414735665233408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68162512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚408175102120474 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68182546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35066537 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414733038632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408177725833110 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122210469 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚70336512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚407955869302784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9300971828224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6765949046298 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚409 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348986053801984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407955842911232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60530464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25464339 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414733038632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54027923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407955869302784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358286997495808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _runtime‚3099.821571487 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6779795930624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54027923 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348986025667584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚401409153074176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414735665233408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777195721216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60530876 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647ª˜ ö2’- _wandbruntime‚3100  _runtime‚3100™– —“‚N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60546290 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35075733 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884406605315533e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414845437280768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408063782616576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35075730 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54042153 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25470949 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 _step‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358381935601664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408063782616576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68180746 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_baè¡›vqtch_end/reserved_bytes.all.peak‚ 23668457472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54042153 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6767778388966 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6781628272640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122242965 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68200806 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358382005924864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414845437280768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60546682 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚408285896350258 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122222899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54042159 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6779028049920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6770376673842 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54042159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚389 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401515519676416 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414842810666496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349078441855488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68200806 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408063809008128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6779028049920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 _runtime‚3100.577034127º±åb '”‡N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122222899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54042159 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6779028049920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6770376673842 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54042159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚389 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401515519676416 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414842810666496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349078441855488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68200806 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408063809008128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6779028049920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 _runtime‚3100.577034127 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60546290 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35075733 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884406605315533e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414845437280768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408063782616576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35075730 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54042153 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25470949 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 _step‚299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358381935601664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408063782616576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68180746 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54042153 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6767778388966 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6781628272640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122242965 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68200806 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358382005924864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414845437280768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60546682 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚408285896350258 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844«cÌtì2•- _wandbruntime‚3100  _runtime‚3100“ÝH]–× 3loss/FBt3/assignment_loss‚0.0021023755349839726 /loss/FRt4/detection_loss‚2.6603956570227942 2loss/SRqqt1/assignment_loss‚1.0039242279948666 1loss/SRqqt4/assignment_loss‚1.571761572163324 1loss/SRqqt4/detection_loss‚1.9800649321809107 .loss/FBt3/detection_loss‚5.668517549832662 -loss/FBt4/detection_loss‚3.24054808417956 /loss/FRt2/detection_loss‚2.9097411491804652 /loss/FRt2/assignment_loss‚3.220983159210947 0loss/FRt4/assignment_loss‚3.0874133649326505 /loss/FBt2/detection_loss‚3.8086491028467813 1loss/FBt1/assignment_loss‚0.09694023147490625 _step‚300 /loss/FRt1/assignment_loss‚1.604455326284681 2loss/SRqqt3/assignment_loss‚2.4444586191858564 /loss/FRt1/detection_loss‚1.7773529976910474 .loss/FRt3/detection_loss‚2.228202793002297 1loss/SRqqt3/detection_loss‚2.2753192016056607 2loss/SRqqt2/assignment_loss‚1.5810475214807824 1loss/SRqqt2/detection_loss‚1.8775370141321963 /loss/FBt1/detection_loss‚0.8780222397587251 _runtime‚3100.578723471 trainer/global_step‚3649 $ _timestamp‚1.788440660534078e+09 /loss/FRt3/assignment_loss‚4.444783493876457 2loss/FBt2/assignment_loss‚0.014107811284096291 1loss/SRqqt1/detection_loss‚0.9937609324231744 %loss/total_loss‚49.85314444385341 1loss/FBt4/assignment_loss‚0.48305508657358587 epoch‚3ˆÎ b—Ü 1loss/FBt1/assignment_loss‚0.09694023147490625 _step‚300 /loss/FRt1/assignment_loss‚1.604455326284681 2loss/SRqqt3/assignment_loss‚2.4444586191858564 /loss/FRt1/detection_loss‚1.7773529976910474 .loss/FRt3/detection_loss‚2.228202793002297 1loss/SRqqt3/detection_loss‚2.2753192016056607 2loss/SRqqt2/assignment_loss‚1.5810475214807824 1loss/SRqqt2/detection_loss‚1.8775370141321963 /loss/FBt1/detection_loss‚0.8780222397587251 _runtime‚3100.578723471 trainer/global_step‚3649 $ _timestamp‚1.788440660534078e+09 /loss/FRt3/assignment_loss‚4.444783493876457 2loss/FBt2/assignment_loss‚0.014107811284096291 1loss/SRqqt1/detection_loss‚0.9937609324231744 %loss/total_loss‚49.85314444385341 1loss/FBt4/assignment_loss‚0.48305508657358587 epoch‚3 3loss/FBt3/assignment_loss‚0.0021023755349839726 /loss/FRt4/detection_loss‚2.6603956570227942 2loss/SRqqt1/assignment_loss‚1.0039242279948666 1loss/SRqqt4/assignment_loss‚1.571761572163324 1loss/SRqqt4/detection_loss‚1.9800649321809107 .loss/FBt3/detection_loss‚5.668517549832662 -loss/FBt4/detection_loss‚3.24054808417956 /loss/FRt2/detection_loss‚2.9097411491804652 /loss/FRt2/assignment_loss‚3.220983159210947 0loss/FRt4/assignment_loss‚3.0874133649326505 /loss/FBt2/detection_loss‚3.8086491028467813¬ßæU“˜‚(:ˆ ØàåÔ¢¬œÉ gpu.0.gpu‚70.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚255.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(zaê0™‚(:¥ ØàåÔôÍ£Ë disk.rbd3.in‚0 disk.dm-0.in‚1724.625 disk.dm-1.out‚ 125.5703125* proc.memory.percent‚16.002607345581055 network.recv‚ 1.134345e+06) disk./.usagePercent‚74.34651388806056% disk./.usageGB‚1107.8167839050293 disk.nvme0n1.in‚1724.625% proc.memory.availableMB‚ 3916.5625 disk.rbd3.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚32832.20654296875$ memory_percent‚76.09519958496094 disk.rbd1.in‚0! disk.rbd2.out‚181.26123046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.533203125& disk.nvme0n1.out‚32832.20654296875( disk.nvme0c0n1.out‚32832.20654296875 disk.rbd2.in‚0 disk.sda.in‚0.5625 cpu‚25.331323310139396 network.sent‚ 8.226861e+06 disk.drbd1777.in‚0 disk.sda.out‚ 125.5703125 disk.sda1.out‚0 disk.sda3.out‚ 125.5703125 disk.nvme0c0n1.in‚1724.625 disk.dm-1.in‚0.5625" proc.memory.rssMB‚ 2621.8671875 disk.sda2.out‚0 disk.sda3.in‚0.5625‘°êÖš‚(:’ çàåÔãäŠÉ gpu.0.gpu‚94.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(œž2›‚(:§ çàåÔ櫳Ë disk.sda3.in‚0.5625# proc.memory.rssMB‚ 2626.45703125 network.sent‚ 8.336127e+06! disk.rbd2.out‚182.27294921875 disk.nvme0c0n1.in‚1724.625( disk.nvme0c0n1.out‚32833.14990234375 cpu‚25.329980302235015 network.recv‚ 1.141151e+06 disk.rbd1.out‚ 1.533203125 disk.rbd3.in‚0) disk./.usagePercent‚74.34651260802289 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 126.0546875 disk.sda2.out‚0 disk.dm-0.in‚1724.625# disk.dm-0.out‚32833.14990234375 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.dm-1.out‚ 126.0546875$ memory_percent‚76.12509727478027$ disk./.usageGB‚1107.816764831543& disk.nvme0n1.out‚32833.14990234375 disk.rbd0.out‚0 disk.dm-1.in‚0.5625( proc.memory.availableMB‚ 3911.6640625) proc.memory.percent‚16.03062152862549 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 126.0546875` ,ê‘œ‚(:† öàåÔúäžÉ gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚257.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“G¿E5‚(:ª öàåÔÚÉ®Ë# disk.dm-0.out‚32833.40771484375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 126.55078125( disk.nvme0c0n1.out‚32833.40771484375# proc.memory.rssMB‚ 2627.04296875 disk.rbd1.out‚ 1.533203125 disk.rbd3.out‚0 disk.nvme0c0n1.in‚1724.625$ memory_percent‚76.12707614898682 network.sent‚ 8.341426e+06 disk.sda.out‚ 126.55078125 disk.rbd3.in‚0 cpu‚25.32865274889312) disk./.usagePercent‚74.36335867179851! disk.rbd2.out‚184.47607421875 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda3.in‚0.5625 proc.cpu.threads‚17 network.recv‚ 1.144932e+06 disk.dm-1.out‚ 126.55078125 disk.nvme0n1.in‚1724.625$ disk./.usageGB‚1108.067783355713 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 3911.33984375) proc.memory.percent‚16.03419780731201 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚1724.625 disk.dm-1.in‚0.5625& disk.nvme0n1.out‚32833.40771484375©Y9Ä2ž- _wandbruntime‚3138  _runtime‚3138ûú̘ÈŸÂ ' lr-AdamW/pg1‚0.00033956263430640353 ' lr-AdamW/pg2‚0.00033956263430640353 trainer/global_step‚3699 $ _timestamp‚1.788440698793036e+09 _runtime‚3138.837739988 _step‚301i¦SÍ Ç ' lr-AdamW/pg1‚0.00033956263430640353 ' lr-AdamW/pg2‚0.00033956263430640353 trainer/global_step‚3699 $ _timestamp‚1.788440698793036e+09 _runtime‚3138.837739988 _step‚301­Š— 2¡- _wandbruntime‚3138  _runtime‚3138”?R§Ð"¢øO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69097598 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420224281349632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413352854569984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35526337 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363032984179200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413583641558682 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25800371 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6871426779648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35526334 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9427479208448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61326248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6871426779648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54739717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9427527702016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420221654748672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚302 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69077564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413352854569984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413352828178432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353605428342784 _runtime‚3138.838800806 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54739717 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420221654748672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413352828178432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69077564 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123837321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25799914 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6857429770906 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 % _timestamp‚1.7884406987938254e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363032907551232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413586265271318 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406726237215744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚3699 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54739723 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123817281 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54739723 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚3ÔÇ .53605456477184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6860028055574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69097598 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420224281349632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413352854569984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35526337 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363032984179200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413583641558682 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25800371 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6871426779648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35526334 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9427479208448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61326248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6871426779648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54739717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9427527702016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚460 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420221654748672 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚302 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69077564®ÆÎÀ2¤- _wandbruntime‚3139  _runtime‚3139Š7 '¥N EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123849751 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406832578390016 $ _timestamp‚1.788440699529089e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413691812075622 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 trainer/global_step‚3699 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69115858 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6873259121664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413460767883776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413460767883776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420334053396992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9430054784000 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54753953 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6861857398450 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69095798 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413460794275328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54753959 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69095798 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69115858 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61342160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353697816396288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6859259113574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54753959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35535533 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353697844530688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6870658898944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363127899314688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9430006303744 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363127822700032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 _step‚303€¨ÆÇ '¦†N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413460794275328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54753959 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69095798 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69115858 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61342160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353697816396288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6859259113574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54753959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35535533 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353697844530688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6870658898944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363127899314688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9430006303744 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363127822700032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 _step‚303 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123849751 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406832578390016 $ _timestamp‚1.788440699529089e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413691812075622 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 trainer/global_step‚3699 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69115858 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6873259121664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413460767883776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413460767883776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420334053396992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9430054784000 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54753953 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6861857398450 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69095798 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6948910495514 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70012650 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 " _timestamp‚1.7884407379361e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425710270864896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6963057628672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55451523 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6960457419264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55451517 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418749813445632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367778661638656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425710270864896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26135640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6951508780182 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6960457419264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6963057628672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418994804709526ýJ(³úO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35986134 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26136084 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6948910495514 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70012650 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 " _timestamp‚1.7884407379361e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425710270864896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6963057628672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55451523 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6960457419264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55451517 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418749813445632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367778661638656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425710270864896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26135640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125464173 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6951508780182 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6960457419264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6963057628672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418994804709526²Ô„<2´- _wandbruntime‚3178  _runtime‚3178ƒ¯Ídµ‚N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412149611675648 _step‚307 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35995330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419102975226674 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70030910 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70010850 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6962289747968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55465753 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 _runtime‚3178.720100931 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9556386516992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 % _timestamp‚1.7884407386749632e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6962289747968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70030910 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6950739838182 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418857779542528 RCDeviceStatsMonitor.on_train_batch_end/in˜ÓN¾¤ active_split_bytes.all.peak‚ 2686470144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55465753 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6964889970688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425822669513216 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6953338123058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412149637103616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62137680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358317219071488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚3749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70010850 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55465759 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367873559205376õð6 '¶‡N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6953338123058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412149637103616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62137680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358317219071488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚3749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70010850 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55465759 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367873559205376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412149611675648 _step‚307 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35995330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419102975226674 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70030910 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70010850 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6962289747968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55465753 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 _runtime‚3178.720100931 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9556386516992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 % _timestamp‚1.7884407386749632e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6962289747968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70030910 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6950739838182 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418857779542528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55465753 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6964889970688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425822669513216 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233³ dv2·- _wandbruntime‚3178  _runtime‚3178þOÇ?W¸Ñ 2loss/SRqqt2/assignment_loss‚1.3225448513930698 epoch‚3 % _timestamp‚1.7884407386770847e+09 1loss/FBt3/assignment_loss‚0.10868472219832863 /loss/FBt4/detection_loss‚2.0873613178730013 1loss/SRqqt2/detection_loss‚1.8927643810803036 .loss/FBt1/detection_loss‚0.641105623528197 .loss/FBt2/detection_loss‚4.524432063102722 %loss/total_loss‚46.46755748362226 2loss/SRqqt3/assignment_loss‚1.9211074424641472 2loss/SRqqt4/assignment_loss‚1.8065768873452075 .loss/FRt3/detection_loss‚2.346894074130708 1loss/SRqqt1/detection_loss‚1.1973666238288085 /loss/SRqqt4/detection_loss‚1.71839350728052 _runtime‚3178.721706286 _step‚308 /loss/FRt1/assignment_loss‚1.537566900253296 0loss/FRt2/assignment_loss‚3.2157905836438023 2loss/SRqqt1/assignment_loss‚1.1641652217755716 1loss/FBt2/assignment_loss‚0.22968260385096073 .loss/FRt2/detection_loss‚2.519301797068396 .loss/FRt4/detection_loss‚2.424785727635026 1loss/FBt1/assignment_loss‚0.22560491240398226 /loss/FBt3/detection_loss‚3.7083639105161033 /loss/FRt3/assignment_loss‚4.546427597870698 /loss/FRt1/detection_loss‚2.0355662357062103 /loss/FRt4/assignment_loss‚2.724595262358586 0loss/FBt4/assignment_loss‚0.1724674234399572 0loss/SRqqt3/detection_loss‚2.396007812874658 trainer/global_step‚3749­ôQÌ\¹Ö 1loss/FBt2/assignment_loss‚0.22968260385096073 .loss/FRt2/detection_loss‚2.519301797068396 .loss/FRt4/detection_loss‚2.424785727635026 1loss/FBt1/assignment_loss‚0.22560491240398226 /loss/FBt3/detection_loss‚3.7083639105161033 /loss/FRt3/assignment_loss‚4.546427597870698 /loss/FRt1/detection_loss‚2.0355662357062103 /loss/FRt4/assignment_loss‚2.724595262358586 0loss/FBt4/assignment_loss‚0.1724674234399572 0loss/SRqqt3/detection_loss‚2.396007812874658 trainer/global_step‚3749 2loss/SRqqt2/assignment_loss‚1.3225448513930698 epoch‚3 % _timestamp‚1.7884407386770847e+09 1loss/FBt3/assignment_loss‚0.10868472219832863 /loss/FBt4/detection_loss‚2.0873613178730013 1loss/SRqqt2/detection_loss‚1.8927643810803036 .loss/FBt1/detection_loss‚0.641105623528197 .loss/FBt2/detection_loss‚4.524432063102722 %loss/total_loss‚46.46755748362226 2loss/SRqqt3/assignment_loss‚1.9211074424641472 2loss/SRqqt4/assignment_loss‚1.8065768873452075 .loss/FRt3/detection_loss‚2.346894074130708 1loss/SRqqt1/detection_loss‚1.1973666238288085 /loss/SRqqt4/detection_loss‚1.71839350728052 _runtime‚3178.721706286 _step‚308 /loss/FRt1/assignment_loss‚1.537566900253296 0loss/FRt2/assignment_loss‚3.2157905836438023 2loss/SRqqt1/assignment_loss‚1.1641652217755716´8몺º‚(:’ £áåÔ†¦ƒÉ gpu.0.gpu‚80.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.557' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.91140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íu‡1»‚(:¦ £áåÔþ¡éË disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚32932.7666015625 disk.sda.out‚ 128.16015625 disk.nvme0n1.in‚1724.625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0$ memory_percent‚76.13084316253662) proc.memory.availableMB‚ 3910.72265625* proc.memory.percent‚16.036701202392578 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0c0n1.in‚1724.625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.533203125! proc.memory.rssMB‚ 2627.453125) disk./.usagePercent‚74.36329031778686$ disk./.usageGB‚1108.066764831543 disk.dm-1.in‚0.5625 disk.sda3.out‚ 128.16015625 cpu‚25.32465738143644 network.sent‚ 8.462178e+06 disk.dm-1.out‚ 128.16015625 disk.sda3.in‚0.5625 network.recv‚ 1.159809e+06 disk.dm-0.in‚1724.625" disk.dm-0.out‚32932.7666015625 disk.rbd1.in‚0 disk.sda.in‚0.5625% disk.nvme0n1.out‚32932.7666015625! disk.rbd2.out‚190.74951171875 disk.sda1.out‚0 disk.sda2.in‚0®ÑE!“¼‚(:ˆ ²áåÔ¤É›É gpu.0.gpu‚74.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZEîó.½‚(:£ ²áåÔõëÐË" disk.dm-0.out‚33030.2392578125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚1724.625 disk.dm-1.out‚ 128.5546875! proc.memory.rssMB‚ 2627.453125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625* proc.memory.percent‚16.036701202392578 cpu‚25.32338145696754 network.sent‚ 8.570558e+06) disk./.usagePercent‚74.36328186953823 disk.sda2.in‚0 disk.sda3.in‚0.5625( proc.memory.availableMB‚ 3910.8046875 disk.sda1.out‚0 disk.rbd0.out‚0! disk.rbd2.out‚192.96826171875 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.sda3.out‚ 128.5546875 network.recv‚ 1.165418e+06% disk./.usageGB‚1108.0666389465332 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625' disk.nvme0c0n1.out‚33030.2392578125 disk.rbd1.out‚ 1.533203125 disk.sda.out‚ 128.5546875 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0% disk.nvme0n1.out‚33030.2392578125$ memory_percent‚76.13034248352051¼1¾‚(:„ ÁáåÔû’—É gpu.0.gpu‚24.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚41.0 gpu.0.powerWatts‚92.01' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œñnr,¿‚(:¡ ÁáåÔƒüæÊ disk.sda.in‚0.5625 disk.sda.out‚ 129.28125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚1724.625! proc.memory.rssMB‚ 2627.453125 cpu‚25.32212812439968 network.sent‚ 8.576443e+06! disk.rbd2.out‚194.99169921875 disk.sda2.in‚0 disk.nvme0c0n1.in‚1724.625 disk.dm-1.in‚0.5625* proc.memory.percent‚16.036701202392578 network.recv‚ 1.169686e+06% disk./.usageGB‚1108.0666389465332 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0& disk.nvme0n1.out‚33030.62060546875# disk.dm-0.out‚33030.62060546875 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚1724.625$ memory_percent‚76.12965106964111 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚33030.62060546875) proc.memory.availableMB‚ 3910.91796875 proc.cpu.threads‚17) disk./.usagePercent‚74.36328186953823 disk.sda3.out‚ 129.28125 disk.rbd0.in‚0 disk.dm-1.out‚ 129.28125“k³t2À- _wandbruntime‚3216  _runtime‚3216or£ÈÁ trainer/global_step‚3799 $ _timestamp‚1.788440776844729e+09 _runtime‚3216.889400589 _step‚309 ' lr-AdamW/pg1‚0.00033581818877593134 ' lr-AdamW/pg2‚0.00033581818877593134ñþSÚÍÂÇ ' lr-AdamW/pg2‚0.00033581818877593134 trainer/global_step‚3799 $ _timestamp‚1.788440776844729e+09 _runtime‚3216.889400589 _step‚309 ' lr-AdamW/pg1‚0.00033581818877593134µç*>2Ã- _wandbruntime‚3216  _runtime‚3216÷ËòMþ'ÄøO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431201513582080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26470805 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362844261827584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36445937 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚424146798712832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26471269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372523945949696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62917206 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56163323 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431198886981120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7054688477696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7052088268288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3216.890378342 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9679684122112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70907668 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62916739 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362844233693184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56163317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56163323 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417360329214976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7054688477696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7042989504790 _step‚310 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7040391220122 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431198886981120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424403344147734 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431201513582080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚424146825104384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚424146825104384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70927702 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372523869321728p&ÂM²ÅýO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62917206 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56163323 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431198886981120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7054688477696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7052088268288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3216.890378342 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9679684122112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70907668 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train}ÄÔQ!_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62916739 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362844233693184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56163317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56163323 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417360329214976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7054688477696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7042989504790 _step‚310 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7040391220122 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431198886981120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424403344147734 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚431201513582080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚424146825104384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚424146825104384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70927702 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372523869321728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431201513582080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26470805 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362844261827584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36445937 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚424146798712832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26471269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372523945949696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¶«Žáó2Æ- _wandbruntime‚3217  _runtime‚3217èo_D'Ç‚N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3217.627745596 _step‚311 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56177559 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70945962 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431311285629440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70925902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9682196321792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9682147841536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7044818847666 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362936649881088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36455130 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7053920596992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127103455 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431311285629440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62932655 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372618846202880 trainer/global_step‚3799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7056520819712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 % _timestamp‚1.7884407775825331e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70925902 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372618769588224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424508890952038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424254764809728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431308659015168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70945962 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424254764809728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56177553 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417466695817216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ňûh 'ȇN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56177559 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70945962 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431311285629440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70925902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9682196321792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9682147841536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7044818847666 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362936649881088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36455130 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7053920596992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127103455 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚431311285629440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62932655 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372618846202880 trainer/global_step‚3799 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7056520819712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 % _timestamp‚1.7884407775825331e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70925902 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372618769588224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424508890952038 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424254764809728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431308659015168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70945962 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424254764809728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56177553 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417466695817216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3217.627745596 _step‚311 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401·N"lH2É- _wandbruntime‚3217  _runtime‚3217åIlZÊÔ /loss/FRt3/assignment_loss‚4.000684319540512 1loss/SRqqt3/detection_loss‚2.2812477786366534 /loss/FBt1/detection_loss‚0.5171134420543448 /loss/FBt4/detection_loss‚2.8852826091978283 epoch‚3 0loss/FRt1/assignment_loss‚1.4331106474233228 2loss/SRqqt2/assignment_loss‚1.9233195044100284 2loss/SRqqt1/assignment_loss‚0.9067669563070947 2loss/SRqqt3/assignment_loss‚1.5320793951793414 -loss/FRt3/detection_loss‚2.32745937829816 1loss/SRqqt4/detection_loss‚1.9211136472721895 trainer/global_step‚3799 _step‚312 /loss/FRt2/detection_loss‚2.4154620844384898 /loss/FBt3/detection_loss‚2.6518140860966275 %loss/total_loss‚43.06397980938629 _runtime‚3217.628943574 0loss/FBt4/assignment_loss‚0.1879809499821729 /loss/FRt2/assignment_loss‚3.257441036079241 1loss/FBt1/assignment_loss‚0.15943405711984476 /loss/FBt3/assignment_loss‚0.039484255714342 /loss/FRt1/detection_loss‚1.7488577844098556 % _timestamp‚1.7884407775843537e+09 1loss/FBt2/assignment_loss‚0.25847537530353293 /loss/FRt4/detection_loss‚2.2594406444292803 .loss/FBt2/detection_loss‚3.083873748779297 0loss/FRt4/assignment_loss‚2.9447736866199055 2loss/SRqqt4/assignment_loss‚1.4537803431351979 1loss/SRqqt1/detection_loss‚0.9511069100652192 0loss/SRqqt2/detection_loss‚1.923877168893814î4 _ËÙ 2loss/SRqqt2/assignment_loss‚1.9233195044100284 2loss/SRqqt1/assignment_loss‚0.9067669563070947 2loss/SRqqt3/assignment_loss‚1.5320793951793414 -loss/FRt3/detection_loss‚2.32745937829816 1loss/SRqqt4/detection_loss‚1.9211136472721895 trainer/global_step‚3799 _step‚312 /loss/FRt2/detection_loss‚2.4154620844384898 /loss/FBt3/detection_loss‚2.6518140860966275 %loss/total_loss‚43.06397980938629 _runtime‚3217.628943574 0loss/FBt4/assignment_loss‚0.1879809499821729 /loss/FRt2/assignment_loss‚3.257441036079241 1loss/FBt1/assignment_loss‚0.15943405711984476 /loss/FBt3/assignment_loss‚0.039484255714342 /loss/FRt1/detection_loss‚1.7488577844098556 % _timestamp‚1.7884407775843537e+09 1loss/FBt2/assignment_loss‚0.25847537530353293 /loss/FRt4/detection_loss‚2.2594406444292803 .loss/FBt2/detection_loss‚3.083873748779297 0loss/FRt4/assignment_loss‚2.9447736866199055 2loss/SRqqt4/assignment_loss‚1.4537803431351979 1loss/SRqqt1/detection_loss‚0.9511069100652192 0loss/SRqqt2/detection_loss‚1.923877168893814 /loss/FRt3/assignment_loss‚4.000684319540512 1loss/SRqqt3/detection_loss‚2.2812477786366534 /loss/FBt1/detection_loss‚0.5171134420543448 /loss/FBt4/detection_loss‚2.8852826091978283 epoch‚3 0loss/FRt1/assignment_loss‚1.4331106474233228¸@ Ž­žÌ‚(:“ ÐáåÔùƒ¨É gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚101.489' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.297800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÐ@ Í‚(:© ÐáåÔÐöÎË disk.sda1.in‚0! proc.memory.rssMB‚ 2627.453125% disk./.usageGB‚1108.0666999816895& disk.nvme0n1.out‚33128.10888671875( disk.nvme0c0n1.out‚33128.10888671875 cpu‚25.320782788873586 disk.sda1.out‚0) disk./.usagePercent‚74.36328596565878 disk.rbd0.out‚0 network.recv‚ 1.17411e+06 disk.rbd1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚33128.10888671875 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 129.59765625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 129.59765625( proc.memory.availableMB‚ 3910.8671875 network.sent‚ 8.582351e+06 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚1724.625* proc.memory.percent‚16.036701202392578 disk.nvme0n1.in‚1724.625 disk.sda3¬Ãå”.in‚0.5625$ memory_percent‚76.12996101379395 disk.sda3.out‚ 129.59765625 disk.rbd1.out‚ 1.533203125! disk.rbd2.out‚198.21435546875ŽgÎ#žÎ‚(:“ ßáåÔœ‹³É gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚241.365' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.272999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ó0Ï‚(:¥ ßáåÔïô¤Ë disk.sda.in‚0.5625 disk.sda.out‚ 130.4296875$ memory_percent‚76.13019943237305( disk./.usagePercent‚74.3632703491992 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚33128.66162109375 disk.sda2.in‚0 cpu‚25.319544839725015 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.drbd1777.out‚0! disk.rbd2.out‚200.23779296875' proc.memory.availableMB‚ 3910.828125& disk.nvme0n1.out‚33128.66162109375 disk.sda3.out‚ 130.4296875 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚33128.66162109375 disk.dm-1.in‚0.5625 disk.sda2.out‚0! proc.memory.rssMB‚ 2627.453125* proc.memory.percent‚16.036701202392578 proc.cpu.threads‚17 network.recv‚ 1.180651e+06 disk.nvme0n1.in‚1724.625 disk.rbd0.in‚0 disk.dm-0.in‚1724.625 disk.dm-1.out‚ 130.4296875 disk.rbd1.in‚0% disk./.usageGB‚1108.0664672851562 disk.nvme0c0n1.in‚1724.625 disk.sda1.in‚0 network.sent‚ 8.691392e+06 disk.rbd1.out‚ 1.533203125‹<“Ђ(:ˆ îáåÔÆÈ€É gpu.0.gpu‚76.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÔœ0Ñ‚(:¥ îáåÔþ‹Ì disk.dm-1.out‚ 130.98828125 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚1724.625% disk.nvme0n1.out‚33129.0048828125 disk.nvme0c0n1.in‚1724.625 disk.dm-1.in‚0.5625 disk.rbd0.out‚0 disk.drbd1777.in‚0) disk./.usagePercent‚74.39696119671275 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125! proc.memory.rssMB‚ 2627.453125 disk.rbd3.out‚0! disk.rbd2.out‚202.45263671875 disk.rbd0.in‚0# memory_percent‚76.1286735534668 cpu‚25.318375874478924 network.recv‚ 1.184293e+06% disk./.usageGB‚1108.5684852600098 disk.rbd2.in‚0' disk.nvme0c0n1.out‚33129.0048828125* proc.memory.percent‚16.036701202392578 disk.dm-0.in‚1724.625" disk.dm-0.out‚33129.0048828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 130.98828125' proc.memory.availableMB‚ 3911.078125 network.sent‚ 8.696803e+06 disk.sda.out‚ 130.98828125 disk.rbd3.in‚06.X_2Ò- _wandbruntime‚3256  _runtime‚3256Š"ø{ÆÓÀ & lr-AdamW/pg2‚0.0003339013613700477 trainer/global_step‚3849 $ _timestamp‚1.788440815982004e+09 _runtime‚3256.026688942 _step‚313 & lr-AdamW/pg1‚0.0003339013613700477»…$ÞËÔÅ _runtime‚3256.026688942 _step‚313 & lr-AdamW/pg1‚0.0003339013613700477 & lr-AdamW/pg2‚0.0003339013613700477 trainer/global_step‚3849 $ _timestamp‚1.788440815982004e+09¹Íï2E2Õ- _wandbruntime‚3256  _runtime‚3256íÆ„^þ'ÖøO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422677413356544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422677387928576 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚429543810371584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7146319326720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9805254083072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 _runtime‚3256.027737127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429543783980032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71822720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63712807 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚377268918585856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 trainer/global_step‚3849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436687503097344 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367463636368384 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56875123 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56875123 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7146319326720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26807070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367463664502784 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7143719117312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429543810371584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36905737 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429543783980032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7134470229398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36905734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71842754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26806599 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71842754 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436690129698304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436687503097344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429811883585942 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7143719117312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63712333 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9805203492352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7131871944730 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 % _timestamp‚1.7884408159827673e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56875117 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71842754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26806599 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71842754 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436690129698304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436687503097344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429811883585942 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7143719117312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63712333 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9805203492352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7131871944730 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 % _timestamp‚1.7884408159827673e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56875117 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9805254083072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 _runtime‚3256.027737127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚429543783980032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71822720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63712807 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚377268918585856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 trainer/global_step‚3849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436687503097344 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367463636368384 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56875123 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56875123 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7146319326720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26807070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367463664502784 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7143719117312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429543810371584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36905737 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429543783980032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7134470229398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573ºO~;2Ø- _wandbruntime‚3256  _runtime‚3256r„X!Ù‚N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429651723685376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436799901745664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚422783754530816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367556052556288 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚436797275131392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429920054103090 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56889353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367556024421888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56889359 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128730307 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884408167211509e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71840954 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63728679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26813302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56889353 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63728232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422783729102848 _step‚315 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429651750076928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429917430390246 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3849 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647Áþ 'Ú‡N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56889353 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63728232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422783729102848 _step‚315 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429651750076928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429917430390246 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3849 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429651723685376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436799901745664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚422783754530816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367556052556288 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚436797275131392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429920054103090 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56889353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367556024421888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56889359 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128730307 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884408167211509e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71840954 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63728679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26813302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0»;lçq2Û- _wandbruntime‚3256  _runtime‚32562š–SÜÍ 0loss/FRt1/assignment_loss‚1.4469078481197357 0loss/FRt4/assignment_loss‚3.2444488721735336 /loss/FBt4/assignment_loss‚0.886614625628751 .loss/FRt4/detection_loss‚2.885353707215365 -loss/FBt3/detection_loss‚6.47251638174057 /loss/FRt2/assignment_loss‚3.245989133252038 1loss/SRqqt4/assignment_loss‚1.812571515639623 0loss/FBt3/assignment_loss‚0.6115927860140801 0loss/SRqqt4/detection_loss‚2.111048162365571 % _timestamp‚1.7884408167231867e+09 2loss/SRqqt2/assignment_loss‚1.8118066683094551 .loss/FRt2/detection_loss‚2.339342345794042 1loss/SRqqt2/detection_loss‚1.7833783328533173 %loss/total_loss‚49.19117501871171 epoch‚3 1loss/SRqqt1/assignment_loss‚0.899928383094569 /loss/FRt1/detection_loss‚1.5549281779516102 1loss/SRqqt3/detection_loss‚2.0833489887024226 0loss/FBt2/assignment_loss‚1.0607090374188763 /loss/FBt4/detection_loss‚1.9849811173402345 _step‚316 /loss/FRt3/assignment_loss‚4.089316629701191 _runtime‚3256.767858833 1loss/FBt1/assignment_loss‚0.16052285716866926 0loss/SRqqt1/detection_loss‚0.997356693012019 .loss/FBt1/detection_loss‚0.735491068696969 trainer/global_step‚3849 2loss/SRqqt3/assignment_loss‚2.2677314191272386 .loss/FRt3/detection_loss‚2.442021932866853 /loss/FBt2/detection_loss‚2.2632683345249722Ñ62XÝÒ %loss/total_loss‚49.19117501871171 epoch‚3 1loss/SRqqt1/assignment_loss‚0.899928383094569 /loss/FRt1/detection_loss‚1.5549281779516102 1loss/SRqqt3/detection_loss‚2.0833489887024226 0loss/FBt2/assignment_loss‚1.0607090374188763 /loss/FBt4/detection_loss‚1.9849811173402345 _step‚316 /loss/FRt3/assignment_loss‚4.089316629701191 _runtime‚3256.767858833 1loss/FBt1/assignment_loss‚0.16052285716866926 0loss/SRqqt1/detection_loss‚0.997356693012019 .loss/FBt1/detection_loss‚0.735491068696969 trainer/global_step‚3849 2loss/SRqqt3/assignment_loss‚2.2677314191272386 .loss/FRt3/detection_loss‚2.442021932866853 /loss/FBt2/detection_loss‚2.2632683345249722 0loss/FRt1/assignment_loss‚1.4469078481197357 0loss/FRt4/assignment_loss‚3.2444488721735336 /loss/FBt4/assignment_loss‚0.886614625628751 .loss/FRt4/detection_loss‚2.885353707215365 -loss/FBt3/detection_loss‚6.47251638174057 /loss/FRt2/assignment_loss‚3.245989133252038 1loss/SRqqt4/assignment_loss‚1.812571515639623 0loss/FBt3/assignment_loss‚0.6115927860140801 0loss/SRqqt4/detection_loss‚2.111048162365571 % _timestamp‚1.7884408167231867e+09 2loss/SRqqt2/assignment_loss‚1.8118066683094551 .loss/FRt2/detection_loss‚2.339342345794042 1loss/SRqqt2/detection_loss‚1.7833783328533173¼z¡ÄÞ‚(:’ ýáåÔ‹çÂÉ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚238.402' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.68039999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Õú“1ß‚(:¦ ýáåÔ  ìË! disk.rbd2.out‚205.48779296875 disk.rbd3.out‚0& disk.nvme0n1.out‚33227.26708984375 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚33227.26708984375) disk./.usagePercent‚74.39682192861412 disk.nvme0n1.in‚1724.625 disk.nvme0c0n1.in‚1724.625 disk.dm-1.out‚ 131.5546875 disk.drbd1777.out‚0 disk.sda2.out‚0# memory_percent‚76.1303186416626( disk.nvme0c0n1.out‚33227.26708984375 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.sda.out‚ 131.5546875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 3910.80859375 disk.sda3.in‚0.5625 disk.rbd1.out‚ 1.533203125* proc.memory.percent‚16.036701202392578% disk./.usageGB‚1108.5664100646973 disk.rbd3.in‚0 cpu‚25.31708344391506 proc.cpu.threads‚17 network.sent‚ 8.702688e+06 network.recv‚ 1.188081e+06 disk.sda3.out‚ 131.5546875 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2627.453125 disk.rbd1.in‚0 disk.rbd2.in‚0KûÓà‚(:ˆ ŒâåÔ׌´É gpu.0.gpu‚79.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚256.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_o·Õ3á‚(:¨ ŒâåÔØ„‘Ì disk.sda3.out‚ 131.90234375( disk.nvme0c0n1.out‚33227.53271484375 disk.rbd1.in‚0$ memory_percent‚76.12967491149902* proc.memory.percent‚16.036701202392578 disk.rbd2.in‚0 disk.sda.in‚0.5625 proc.cpu.threads‚17% disk./.usageGB‚1108.5664100646973& disk.nvme0n1.out‚33227.53271484375 disk.rbd0.in‚0 disk.rbd1.out‚ 1.533203125( proc.memory.availableMB‚ 3910.9140625! proc.memory.rssMB‚ 2627.453125 network.sent‚ 8.81131e+06 disk.nvme0c0n1.in‚1724.625# disk.dm-0.out‚33227.53271484375 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0! disk.rbd2.out‚210.74951171875 disk.sda1.in‚0 cpu‚25.31594440435319 disk.rbd3.in‚0 disk.nvme0n1.in‚1724.625 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 131.90234375 disk.sda.out‚ 131.90234375 network.recv‚ 1.194642e+06 disk.sda3.in‚0.5625 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0 disk.sda2.out‚0) disk./.usagePercent‚74.39682192861412A÷€2â- _wandbruntime‚3295  _runtime‚3295Uµá«ÇãÁ trainer/global_step‚3899 % _timestamp‚1.7884408550305288e+09 _runtime‚3295.075210445 _step‚317 & lr-AdamW/pg1‚0.0003319553280991081 & lr-AdamW/pg2‚0.0003319553280991081OíO;ÌäÆ _step‚317 & lr-AdamW/pg1‚0.0003319553280991081 & lr-AdamW/pg2‚0.0003319553280991081 trainer/global_step‚3899 % _timestamp‚1.7884408550305288e+09 _runtime‚3295.075210445½ºÓš2å- _wandbruntime‚3295  _runtime‚3295£’­ý'æ÷O N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130344729 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27143377 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130324689 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57586917 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37365534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72737772 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382014084552192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225950954006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442178745814528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64508445 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7223352669338 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442178745814528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372083067177984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚435220423024150 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 % _timestamp‚1.7884408550313022e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372083039043584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434940769247232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442176119213568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434940769247232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435217799311514 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72757806 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57586923 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 _step‚318 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434940795638784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7235349966336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37365537 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7237950175744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27142911 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7237950175744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72757806 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434940795638784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9931017374208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427994472070144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442176119213568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9930968880640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚435220423024150 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 % _timestamp‚1.7884408550313022e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372083039043584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚434940769247232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442176119213568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434940769247232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435217799311514 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72757806 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57586923 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 _step‚318 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434940795638784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7235349966336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37365537 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7237950175744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27142911 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7237950175744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72757806 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434940795638784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¾8'Ñä2è- _wandbruntime‚3295  _runtime‚3295r¾Óý'éN OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 trainer/global_step‚3899 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9933524096000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9933475615744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372175427097088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37374733 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442288517861888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382108979327488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442285891247616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428100787816448 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27150044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57601159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64524777 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7239782517760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _step‚319 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72756006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57601159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57601153 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64524332 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚435048708952576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27149602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788440855772413e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37374730 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435048735344128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7239782517760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7237182295040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57601153 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442288517861888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚435048708952576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚435048735344128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72776066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435328593541298 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F 'ê†N EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27150044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57601159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64524777 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7239782517760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _step‚319 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72756006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57601159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57601153 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64524332 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚435048708952576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27149602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788440855772413e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37374730 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435048735344128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7239782517760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7237182295040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57601153 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442288517861888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚435048708952576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚435048735344128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72776066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435328593541298 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 trainer/global_step‚3899 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9933524096000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9933475615744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚372175427097088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37374733 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130357159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442288517861888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382108979327488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442285891247616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428100787816448¿LPH®2ë- _wandbruntime‚3295  _runtime‚3295¦ÌÜYìÓ 0loss/FRt1/assignment_loss‚1.5739312039481268 .loss/FRt2/detection_loss‚2.414582139679364 2loss/SRqqt4/assignment_loss‚1.4908195795381771 1loss/FBt3/assignment_loss‚0.03699266415141109 .loss/FRt4/detection_loss‚2.529345592347587 epoch‚3 _step‚320 0loss/FRt2/assignment_loss‚2.9718661294096993 0loss/FRt3/assignment_loss‚4.3238502740859985 /loss/FBt2/assignment_loss‚0.609746678173542 0loss/SRqqt3/detection_loss‚2.493949075539907 0loss/SRqqt4/detection_loss‚2.012459852239665 % _timestamp‚1.7884408557743564e+09 .loss/FRt1/detection_loss‚1.501073138029487 /loss/FRt3/detection_loss‚3.2597505385461063 /loss/FBt1/detection_loss‚0.5770741603449436 /loss/FBt2/detection_loss‚3.0897088766098024 2loss/SRqqt2/assignment_loss‚1.7990657101074854 1loss/FBt1/assignment_loss‚0.08523573194254944 1loss/SRqqt1/detection_loss‚0.9984705874029501 /loss/FBt3/detection_loss‚2.7515132096078663 _runtime‚ 3295.81893025 2loss/SRqqt1/assignment_loss‚0.9897209354273883 0loss/FBt4/assignment_loss‚0.1524535829667002 &loss/total_loss‚45.148711240018656 0loss/FRt4/assignment_loss‚3.4740581890431845 1loss/SRqqt3/assignment_loss‚1.689035181204478 1loss/SRqqt2/detection_loss‚1.9830340352323321 .loss/FBt4/detection_loss‚2.340974174439907 trainer/global_step‚3899¼î K^íØ 1loss/SRqqt3/assignment_loss‚1.689035181204478 1loss/SRqqt2/detection_loss‚1.9830340352323321 .loss/FBt4/detection_loss‚2.340974174439907 trainer/global_step‚3899 0loss/FRt1/assignment_loss‚1.5739312039481268 .loss/FRt2/detection_loss‚2.414582139679364 2loss/SRqqt4/assignment_loss‚1.4908195795381771 1loss/FBt3/assignment_loss‚0.03699266415141109 .loss/FRt4/detection_loss‚2.529345592347587 epoch‚3 _step‚320 0loss/FRt2/assignment_loss‚2.9718661294096993 0loss/FRt3/assignment_loss‚4.3238502740859985 /loss/FBt2/assignment_loss‚0.609746678173542 0loss/SRqqt3/detection_loss‚2.493949075539907 0loss/SRqqt4/detection_loss‚2.012459852239665 % _timestamp‚1.7884408557743564e+09 .loss/FRt1/detection_loss‚1.501073138029487 /loss/FRt3/detection_loss‚3.2597505385461063 /loss/FBt1/detection_loss‚0.5770741603449436 /loss/FBt2/detection_loss‚3.0897088766098024 2loss/SRqqt2/assignment_loss‚1.7990657101074854 1loss/FBt1/assignment_loss‚0.08523573194254944 1loss/SRqqt1/detection_loss‚0.9984705874029501 /loss/FBt3/detection_loss‚2.7515132096078663 _runtime‚ 3295.81893025 2loss/SRqqt1/assignment_loss‚0.9897209354273883 0loss/FBt4/assignment_loss‚0.1524535829667002 &loss/total_loss‚45.148711240018656 0loss/FRt4/assignment_loss‚3.4740581890431845À,¼=‘î‚(:† ›âåÔõ©É gpu.0.gpu‚59.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.44' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¹¤Q-ï‚(:¢ ›âåÔÃæþË disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125' proc.memory.availableMB‚ 3910.796875% disk./.usageGB‚1108.5664100646973 disk.rbd3.in‚0# disk.dm-0.out‚33228.46142578125 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚76.13039016723633# proc.memory.rssMB‚ 2628.19921875& disk.nvme0n1.out‚33228.46142578125 network.recv‚ 1.199164e+06! disk.rbd2.out‚214.85888671875 disk.sda.out‚ 132.453125* proc.memory.percent‚16.041254997253418 disk.sda3.in‚0.5625 disk.dm-0.in‚1724.625 disk.dm-1.out‚ 132.453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚1724.625 cpu‚25.3147478541846 disk.sda3.out‚ 132.453125 network.sent‚ 8.81754e+06) disk./.usagePercent‚74.39682192861412 disk.sda2.in‚0 disk.nvme0n1.in‚1724.625( disk.nvme0c0n1.out‚33228.46142578125 disk.dm-1.in‚0.5625GîPh“ð‚(:ˆ ªâåÔ­üÄÈ gpu.0.gpu‚56.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ú5@4ñ‚(:© ªâåÔ¯¢ÖË network.sent‚ 8.92572e+06% disk./.usageGB‚1108.5664100646973 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625 disk.drbd1777.in‚0 network.recv‚ 1.204946e+06 disk.sda2.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚33325.95751953125 disk.drbd1777.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2628.19921875* proc.memory.percent‚16.041254997253418& disk.nvme0n1.out‚33325.95751953125 disk.nvme0c0n1.in‚1724.625 disk.sda1.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚33325.95751953125 disk.dm-0.in‚1724.625 cpu‚25.313489765331838) disk./.usagePercent‚74.39682192861412 disk.sda2.out‚0 disk.sda3.out‚ 132.92578125 disk.dm-1.out‚ 132.92578125 disk.rbd1.out‚ 1.533203125 disk.sda.in‚0.5625 disk.sda1.in‚0& proc.memory.availableMB‚ 3910.53125 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.rbd0.out‚0$ memory_percent‚76.13201141357422! disk.rbd2.out‚222.14404296875 disk.sda.out‚ 132.92578125vÚÓò‚(:’ ¹âåÔꯟÉ gpu.0.gpu‚69.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.449' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.48980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì1ÒÈó‚(:¢ ¹âåÔÕðÑË network.sent‚ 8.931695e+06 cpu‚25.31223818014512 disk.rbd1.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚33326.5419921875# proc.memory.rssMB‚ 2628.19921875* proc.memory.percent‚16.041254997253418% disk./.usageGB‚1108.5664138793945 disk.sda.out‚ 133.359375 disk.rbd3.out‚0 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 133.359375 disk.nvme0c0n1.in‚1724.625 disk.drbd1777.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚33326.5419921875 proc.cpu.threads‚17 disk.sda3.in‚0.5625# memory_percent‚76.1305570602417 disk.rbd2.in‚0! disk.rbd2.out‚225.17919921875) disk./.usagePercent‚74.39682218462166 disk.rbd1.out‚ 1.533203125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚1724.625 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 3910.76953125' disk.nvme0c0n1.out‚33326.5419921875 netwoê9ã erk.recv‚ 1.209443e+06 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 133.359375 disk.drbd1777.in‚0rÅ)²2ô- _wandbruntime‚3334  _runtime‚3334tÒd“ÇõÁ & lr-AdamW/pg1‚0.0003299804945135294 & lr-AdamW/pg2‚0.0003299804945135294 trainer/global_step‚3949 % _timestamp‚1.7884408942573125e+09 _runtime‚3334.301954668 _step‚321¼¬ÌöÆ _step‚321 & lr-AdamW/pg1‚0.0003299804945135294 & lr-AdamW/pg2‚0.0003299804945135294 trainer/global_step‚3949 % _timestamp‚1.7884408942573125e+09 _runtime‚3334.301954668Á²°ø2÷- _wandbruntime‚3334  _runtime‚3334Ow„(øúO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58298723 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚447667361930752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3334.303024735 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7326980815360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386758958562816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10056516844032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131951541 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447667361930752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73652824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚440626338749722 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433311530783744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73672858 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376702469853184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7329581024768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73652824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7329581024768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131971581 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10056567434752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58298723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73672858 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386759037287936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440628962462358 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37825334 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27478819 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 trainer/global_step‚3949 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447664735329792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433311505355776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 % _timestamp‚1.7884408942581604e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7326980815360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65303680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27478346 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447664735329792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433311505355776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 % _timestamp‚1.7884408942581604e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7326980815360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65303680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27478346 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58298723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73672858 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386759037287936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440628962462358 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37825334 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27478819 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 trainer/global_step‚3949 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73691118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7328813144064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65320066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65319596 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447777133978112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433417846530048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7316662736614 _runtime‚3335.042938646 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7328813144064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440737132979506 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73671058 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440445694219776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73671058 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7331413366784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386853845938688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37834533 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386853924650496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376794829772288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7331413366784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37834530 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58312953 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73691118 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440445720611328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58312953 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440445720611328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447777133978112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433417871958016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7319261021490 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376794857906688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚440734509266662 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10059016166400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58312959 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58312959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27485533xÞÕ¢ü‰N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37834530 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58312953 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73691118 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freedžÁ8"ò ‚ 131984011 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440445720611328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58312953 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440445720611328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447777133978112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433417871958016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7319261021490 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376794857906688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚440734509266662 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10059016166400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58312959 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58312959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27485533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73691118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7328813144064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65320066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65319596 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447777133978112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433417846530048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7316662736614 _runtime‚3335.042938646 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7328813144064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440737132979506 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73671058 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440445694219776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73671058 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7331413366784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386853845938688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37834533 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386853924650496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376794829772288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7331413366784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074ïŒöŽ2ý- _wandbruntime‚3335  _runtime‚3335•K)VþÐ 1loss/SRqqt2/detection_loss‚1.9510889084388812 .loss/FBt4/detection_loss‚2.273698389530182 %loss/total_loss‚42.54943211401881 /loss/FRt1/detection_loss‚1.7212606767813365 .loss/FRt2/detection_loss‚2.474157038985229 0loss/FRt2/assignment_loss‚2.8698604019676766 /loss/SRqqt2/assignment_loss‚1.5500073916434 1loss/SRqqt4/assignment_loss‚1.627724783609108 0loss/FBt4/assignment_loss‚0.7406947241785625 0loss/FBt1/assignment_loss‚0.2154980672892208 0loss/FBt3/assignment_loss‚0.5443222008814866 2loss/SRqqt3/assignment_loss‚2.1707570925354958 .loss/FBt2/detection_loss‚2.093467593193054 trainer/global_step‚3949 % _timestamp‚1.7884408949998772e+09 0loss/FRt4/assignment_loss‚3.3200294959545134 .loss/FRt3/detection_loss‚2.218159507621426 epoch‚3 1loss/SRqqt4/detection_loss‚1.8877315025548547 0loss/FRt3/assignment_loss‚3.5459712879224257 1loss/SRqqt1/assignment_loss‚1.007957353909862 1loss/SRqqt1/detection_loss‚1.0957451466884878 1loss/FBt2/assignment_loss‚0.29354217602910165 /loss/FRt4/detection_loss‚2.2768988746404646 /loss/FBt1/detection_loss‚0.5495103286857439 0loss/FRt1/assignment_loss‚1.8947873594714146 1loss/SRqqt3/detection_loss‚1.9850993803361567 .loss/FBt3/detection_loss‚2.241462431170724 _runtime‚3335.044451176 _step‚324Ou(Æ[ÿÕ 1loss/FBt2/assignment_loss‚0.29354217602910165 /loss/FRt4/detection_loss‚2.2768988746404646 /loss/FBt1/detection_loss‚0.5495103286857439 0loss/FRt1/assignment_loss‚1.8947873594714146 1loss/SRqqt3/detection_loss‚1.9850993803361567 .loss/FBt3/detection_loss‚2.241462431170724 _runtime‚3335.044451176 _step‚324 1loss/SRqqt2/detection_loss‚1.9510889084388812 .loss/FBt4/detection_loss‚2.273698389530182 %loss/total_loss‚42.54943211401881 /loss/FRt1/detection_loss‚1.7212606767813365 .loss/FRt2/detection_loss‚2.474157038985229 0loss/FRt2/assignment_loss‚2.8698604019676766 /loss/SRqqt2/assignment_loss‚1.5500073916434 1loss/SRqqt4/assignment_loss‚1.627724783609108 0loss/FBt4/assignment_loss‚0.7406947241785625 0loss/FBt1/assignment_loss‚0.2154980672892208 0loss/FBt3/assignment_loss‚0.5443222008814866 2loss/SRqqt3/assignment_loss‚2.1707570925354958 .loss/FBt2/detection_loss‚2.093467593193054 trainer/global_step‚3949 % _timestamp‚1.7884408949998772e+09 0loss/FRt4/assignment_loss‚3.3200294959545134 .loss/FRt3/detection_loss‚2.218159507621426 epoch‚3 1loss/SRqqt4/detection_loss‚1.8877315025548547 0loss/FRt3/assignment_loss‚3.5459712879224257 1loss/SRqqt1/assignment_loss‚1.007957353909862 1loss/SRqqt1/detection_loss‚1.0957451466884878ÄÔÍ\už€‚(:“ ÈâåÔÿµ¾É gpu.0.gpu‚70.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚261.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.217800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ–yÅ.‚(:£ ÈâåÔ¾ÏòË) disk./.usagePercent‚74.39682218462166 disk.dm-1.out‚ 133.83984375$ memory_percent‚76.13239288330078 cpu‚25.310993492881362 network.sent‚ 8.937445e+06 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1108.5664138793945% disk.nvme0n1.out‚33424.0537109375' disk.nvme0c0n1.out‚33424.0537109375 disk.drbd1777.out‚0 disk.sda.out‚ 133.83984375& proc.memory.availableMB‚ 3910.46875 disk.sda3.out‚ 133.83984375 disk.nvme0c0n1.in‚1724.625 disk.rbd1.in‚0* proc.memory.percent‚16.041254997253418 disk.rbd0.in‚0" disk.dm-0.out‚33424.0537109375 disk.rbd1.out‚ 1.533203125 disk.sda2.out‚0# proc.memory.rssMB‚ 2628.19921875 proc.cpu.threads‚17 network.recv‚ 1.213461e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 225.3359375 disk.nvme0n1.in‚1724.625 disk.dm-0.in‚1724.625 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0µÉ‚‚(:“ ×âåÔŦÉ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚146.272' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.254399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É¢'ƒ‚(:œ ×âåÔë–¯Ë disk.dm-0.out‚33424.40234375 proc.cpu.threads‚17% disk./.usageGB‚1108.5646324157715 disk.rbd2.out‚ 225.85546875* proc.memory.percent‚16.041254997253418 disk.sda1.in‚0 network.sent‚ 9.04679e+06 disk.sda3.in‚0.5625 disk.nvme0c0n1.in‚1724.625 disk.dm-1.out‚ 134.3515625 disk.drbd1777.out‚0 cpu‚25.309845979035483 disk.nvme0n1.in‚1724.625 disk.sda3.out‚ 134.3515625 disk.rbd0.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda.in‚0.5625 disk.sda.out‚ 134.3515625 disk.sda2.out‚0) disk./.usagePercent‚74.39670262910316 disk.drbd1777.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 3910.42578125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2628.19921875 disk.sda1.out‚0 disk.rbd3.out‚0# disk.nvme0n1.out‚33424.40234375 disk.dm-1.in‚0.5625 disk.rbd3.in‚0% disk.nvme0c0n1.out‚33424.40234375 disk.dm-0.in‚1724.625 disk.rbd2.in‚0 disk.sda2.in‚0# memory_percent‚76.1326551437378 network.recv‚ 1.220637e+06ÐÂðû2„- _wandbruntime‚3373  _runtime‚3373‚"GºÇ…Á & lr-AdamW/pg2‚0.0003279772721656711 trainer/global_step‚3999 % _timestamp‚1.7884409334954553e+09 _runtime‚3373.540076607 _step‚325 & lr-AdamW/pg1‚0.0003279772721656711°½c,Ì†Æ & lr-AdamW/pg1‚0.0003279772721656711 & lr-AdamW/pg2‚0.0003279772721656711 trainer/global_step‚3999 % _timestamp‚1.7884409334954553e+09 _runtime‚3373.540076607 _step‚325Ťµi±2‡- _wandbruntime‚3373  _runtime‚3373¨”ïª(ˆúO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59010523 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391503959882240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381321844393984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59010523 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38285137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10182115488256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445734766173184 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59010517 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66099904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27814770 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7406314118554 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38285134 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381321872528384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445734766173184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7418611664384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438628589497344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74567876 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453153351446016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚446037501900566 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884409334961545e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74587910 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66100395 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391504038607360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚3373.540967045 _step‚326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7421211873792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74587910 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445734739781632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74567876 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438628564069376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7421211873792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248­ÛžUA‰ÿO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59010523 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38285137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10182115488256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445734766173184 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59010517 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66099904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27814770 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7406314118554 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38285134 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381321872528384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445734766173184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7418611664384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438628589497344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74567876 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453153351446016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 366520320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚446037501900566 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884409334961545e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74587910 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66100395 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391504038607360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚3373.540967045 _step‚326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7421211873792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74587910 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445734739781632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74567876 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438628564069376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7421211873792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/se‡ÂÐUÄ gment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59010523 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391503959882240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381321844393984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376Ær°2Š- _wandbruntime‚3374  _runtime‚3374376r '‹„N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59024753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445842705878528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74606170 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381414260581888 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66115783 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚381414232447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38294333 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27821453 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚446145672417714 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚453265750094336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438734905243648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59024759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438734930671616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7420443993088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445842679486976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 trainer/global_step‚3999 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74586110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66116243 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27821910 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10184665827840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59024759 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7423044215808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚3374.280697825 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10184613153280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚391598845600768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7408143461222 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568úQ½'Œ‰N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7423044215808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74606170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7420443993088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74586110 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445842705878528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391598926409728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66115783 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚381414232447488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38294333 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27821453 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚446145672417714 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚453265750094336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438734905243648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59024759 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438734930671616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7420443993088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445842679486976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 trainer/global_step‚3999 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74586110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66116243 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27821910 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10184665827840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59024759 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7423044215808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚3374.280697825 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10184613153280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚391598845600768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7408143461222 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59024753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445842705878528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74606170 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381414260581888 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451443417626138 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135225285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38744937 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7512842722816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59722317 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7500393127830 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 _runtime‚3412.979770008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458641967562240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7510242513408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMon3_ [âitor.on_train_batch_start/inactive_split_bytes.all.freed‚396248459576832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443945648210944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75482928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75502962 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7497794843162 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75502962 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396248538301952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443945622782976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10307263098368 trainer/global_step‚4049 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75482928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451131725048832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7512842722816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7510242513408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451131751440384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59722323 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66895701 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385941275203584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59722323 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66895701 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385941275203584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451443417626138 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135225285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38744937 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7512842722816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59722317 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7500393127830 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 _runtime‚3412.979770008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458641967562240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7510242513408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396248459576832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚443945648210944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75482928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75502962 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7497794843162 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75502962 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚396248538301952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚443945622782976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10307263098368 trainer/global_step‚4049 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960Ê6ùY2œ- _wandbruntime‚3413  _runtime‚3413/Äbq '„N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28156956 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458751739596288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386033635122688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66911086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28157406 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451239691145728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451239691145728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396343430384128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451554211855922 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _runtime‚3413.726864914 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38754133 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386033663257088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10309718646784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451239664754176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59736553 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7514675064832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458754366210560 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚4049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458754366210560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7512074842112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884409736817796e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75521222 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75501162 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7512074842112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7499624185830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7514675064832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 _step‚331 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38754130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10309767127040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59736559 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396343353769472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59736553§ÅF¼ž‰N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451239691145728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451239691145728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396343430384128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 366520320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451554211855922 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 _runtime‚3413.726864914 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38754133 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386033663257088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10309718646784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451239664754176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59736553 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7514675064832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes,õ8S.all.allocated‚458754366210560 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚4049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458754366210560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7512074842112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884409736817796e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75521222 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75501162 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7512074842112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7499624185830 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7514675064832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 _step‚331 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38754130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10309767127040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59736559 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396343353769472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59736553 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28156956 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458751739596288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386033635122688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66911086 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28157406 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ËBŽÀH2Ÿ- _wandbruntime‚3413  _runtime‚3413¿bÝU Ï 0loss/FRt3/assignment_loss‚3.6218004152178764 0loss/FBt1/assignment_loss‚0.1092625943624953 1loss/FBt3/assignment_loss‚0.22189430426806211 /loss/FRt4/detection_loss‚2.4239381247240566 1loss/SRqqt2/detection_loss‚1.9351040349765258 0loss/SRqqt4/detection_loss‚1.898391490181287 .loss/FBt2/detection_loss‚2.966837301850319 epoch‚3 .loss/FRt1/detection_loss‚1.861591403838247 0loss/FRt1/assignment_loss‚1.7820271781899713 /loss/FRt4/assignment_loss‚3.455500605313674 2loss/SRqqt3/assignment_loss‚1.7788314722478389 2loss/SRqqt4/assignment_loss‚1.3367638225026894 % _timestamp‚1.7884409736839604e+09 1loss/FBt2/assignment_loss‚0.32038885983638465 0loss/SRqqt3/detection_loss‚2.302468165755272 %loss/total_loss‚46.50175011743765 _runtime‚3413.728607578 /loss/FRt2/assignment_loss‚3.663549215328403 .loss/FRt2/detection_loss‚2.656251277865433 /loss/FRt3/detection_loss‚2.6224595811218046 .loss/FBt4/detection_loss‚2.629422989758578 0loss/SRqqt1/detection_loss‚1.022538850366166 .loss/FBt3/detection_loss‚4.145441810290019 trainer/global_step‚4049 1loss/SRqqt1/assignment_loss‚1.228185978972099 1loss/SRqqt2/assignment_loss‚1.541197411309589 0loss/FBt4/assignment_loss‚0.3648254019391185 /loss/FBt1/detection_loss‚0.6130778272217423 _step‚332õY÷ÊZ¡Ô .loss/FRt2/detection_loss‚2.656251277865433 /loss/FRt3/detection_loss‚2.6224595811218046 .loss/FBt4/detection_loss‚2.629422989758578 0loss/SRqqt1/detection_loss‚1.022538850366166 .loss/FBt3/detection_loss‚4.145441810290019 trainer/global_step‚4049 1loss/SRqqt1/assignment_loss‚1.228185978972099 1loss/SRqqt2/assignment_loss‚1.541197411309589 0loss/FBt4/assignment_loss‚0.3648254019391185 /loss/FBt1/detection_loss‚0.6130778272217423 _step‚332 0loss/FRt3/assignment_loss‚3.6218004152178764 0loss/FBt1/assignment_loss‚0.1092625943624953 1loss/FBt3/assignment_loss‚0.22189430426806211 /loss/FRt4/detection_loss‚2.4239381247240566 1loss/SRqqt2/detection_loss‚1.9351040349765258 0loss/SRqqt4/detection_loss‚1.898391490181287 .loss/FBt2/detection_loss‚2.966837301850319 epoch‚3 .loss/FRt1/detection_loss‚1.861591403838247 0loss/FRt1/assignment_loss‚1.7820271781899713 /loss/FRt4/assignment_loss‚3.455500605313674 2loss/SRqqt3/assignment_loss‚1.7788314722478389 2loss/SRqqt4/assignment_loss‚1.3367638225026894 % _timestamp‚1.7884409736839604e+09 1loss/FBt2/assignment_loss‚0.32038885983638465 0loss/SRqqt3/detection_loss‚2.302468165755272 %loss/total_loss‚46.50175011743765 _runtime‚3413.728607578 /loss/FRt2/assignment_loss‚3.663549215328403ÌtÉR?¢‚(:´ “ãåÔý›šË% disk./.usageGB‚1108.6039924621582% disk.nvme0n1.out‚33563.7197265625 disk.rbd0.out‚0 disk.dm-0.in‚ 1724.62890625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1724.62890625 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125* proc.memory.percent‚16.041254997253418' disk.nvme0c0n1.out‚33563.7197265625" disk.dm-0.out‚33563.7197265625 disk.dm-1.out‚ 136.1953125 disk.drbd1777.out‚0 network.recv‚ 1.238996e+06# disk.nvme0c0n1.in‚ 1724.62890625 disk.rbd2.in‚0 disk.sda.out‚ 136.1953125 disk.sda2.out‚0( proc.memory.availableMB‚ 2036.2734375 disk.sda.in‚0.5625 cpu‚25.30532542570083 network.sent‚ 9.171898e+06) disk./.usagePercent‚74.39934411484145 disk.sda3.out‚ 136.1953125! disk.rbd2.out‚226.36083984375 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2628.19921875 disk.drbd1777.in‚0$ memory_percent‚87.57157325744629ž©¹|“£‚(:ˆ “ãåÔɯ¦Ë gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚226.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.2446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì øT’¤‚(:‡ ¢ãåÔþ®–É gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚46.0 gpu.0.powerWatts‚261.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.307 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢g0 @¥‚(:µ ¢ãåÔô¨ëË disk.sda.out‚ 136.4921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 2036.48046875 proc.cpu.threads‚17 disk.rbd1.out‚ 1.533203125 disk.rbd2.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚33661.2509765625$ disk./.usageGB‚1108.604103088379 disk.sda.in‚0.5625% disk.nvme0n1.out‚33661.2509765625 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚87.57030963897705# proc.memory.rssMB‚ 2628.19921875 network.recv‚ 1.244946e+06 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1724.62890625 disk.dm-1.out‚ 136.4921875 network.sent‚ 9.279883e+06 disk.sda3.out‚ 136.4921875 disk.rbd0.out‚0* proc.memory.percent‚16.041254997253418 cpu‚25.304281628092593 disk.drbd1777.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1724.62890625 disk.dm-0.in‚ 1724.62890625" disk.dm-0.out‚33661.2509765625) disk./.usagePercent‚74.39935153905995! disk.rbd2.out‚226.36083984375 <@¿“¦‚(:ˆ ±ãåÔÑÈ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚227.826' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.5652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xœA§‚(:¶ ±ãåÔû¤Ë network.sent‚ 9.285943e+06 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 137.08984375* proc.memory.percent‚16.041254997253418$ disk./.usageGB‚1108.604103088379" disk.dm-0.out‚33662.3251953125 disk.drbd1777.in‚0 disk.sda.out‚ 137.08984375 disk.sda2.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚33662.3251953125 disk.rbd1.out‚ 1.533203125) disk./.usagePercent‚74.39935153905995 disk.sda2.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.62890625 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 2036.23046875! disk.rbd2.out‚226.42333984375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚33662.3251953125 disk.rbd1.in‚0# memory_percent‚87.5718355178833 network.recv‚ 1.249464e+06 disk.dm-1.out‚ 137.08984375# proc.memory.rssMB‚ 2628.19921875 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1724.62890625 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1724.62890625 cpu‚25.30312122967403 proc.cpu.threads‚17çîD2¨- _wandbruntime‚3452  _runtime‚3452_”›]É©Ã % _timestamp‚1.7884410126596394e+09 _runtime‚3452.704273962 _step‚333 ' lr-AdamW/pg1‚0.00032388733688642986 ' lr-AdamW/pg2‚0.00032388733688642986 trainer/global_step‚4099Oå ÎªÈ trainer/global_step‚4099 % _timestamp‚1.7884410126596394e+09 _runtime‚3452.704273962 _step‚333 ' lr-AdamW/pg1‚0.00032388733688642986 ' lr-AdamW/pg2‚0.00032388733688642986Ídw2«- _wandbruntime‚3452  _runtime‚3452›c(¬úO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60434117 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456528710316032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390560649744384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10432997180416 % _timestamp‚1.7884410126603382e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7604473571840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28485229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76418014 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464130583678464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67690432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28485695 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456528736707584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60434117 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464130583678464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39204737 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚334 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464133210279424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7601873362432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7601873362432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7591873852438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464133210279424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136852137 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67689963 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10432948686848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449262681496576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60434123 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456528710316032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7604473571840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456851957064346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39204734 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390560649744384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10432997180416 % _timestamp‚1.7884410126603382e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7604473571840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28485229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76418014 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464130583678464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67690432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28485695 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456528736707584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60434117 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464130583678464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39204737 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚334 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464133210279424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7601873362432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7601873362432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7591873852438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464133210279424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136852137 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67689963 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10432948686848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449262681496576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60434123 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456528710316032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7604473571840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456851957064346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39204734 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456636650021376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456636676412928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76436274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448353 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401088488243200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464242982326784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456960127581286 _runtime‚3453.465019019 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390653065932288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28492388 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456962751294130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 trainer/global_step‚4099 _step‚335 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67706321 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67705855 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76416214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456636650021376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449369048098816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449369022670848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401088564857856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456636676412928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10435450445312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10435498925568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39213933 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390653037797888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ÍÆ«z'°‰N % _timestamp‚1.7884410134198477e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448359 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76436274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448353 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401088488243200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464242982326784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456960127581286 _runtime‚3453.465019019 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390653065932288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28492388 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456962751294130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 trainer/global_step‚4099 _step‚335 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67706321 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67705855 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76416214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456636650021376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449369048098816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449369022670848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401088564857856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456636676412928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10435450445312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10435498925568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39213933 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390653037797888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76416214 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456636650021376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456636676412928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469621826395648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚475 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405738211631616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462263120215190 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469619199794688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395180080553984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28819599 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7683354577046 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469619199794688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461925721974784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7693504211456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77333066 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77313032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469621826395648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚475 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405738211631616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462263120215190 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469619199794688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395180080553984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28819599 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7683354577046 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469619199794688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461925721974784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7693504211456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77333066 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77313032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68500028 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7685183919922 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚454686081384448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7682585635046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68500496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462033635288576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7695336540160 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462371290732338 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7695336540160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39673730 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10560593907712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462033661680128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 % _timestamp‚1.7884410531999662e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61160159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77351326 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61160153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77351326 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77331266 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 _runtime‚3493.245186889 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚469728971828736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77331266 trainer/global_step‚4149 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7697936762880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469731598443008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138491419 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096‰-7¬>À‰N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚468 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68500028 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7685183919922 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚454686081384448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7682585635046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68500496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462033635288576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7695336540160 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462371290732338 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7695336540160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39673730 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10560593907712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462033661680128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 % _timestamp‚1.7884410531999662e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61160159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77351326 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61160153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77351326 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77331266 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 _runtime‚3493.245186889 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚469728971828736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77331266 trainer/global_step‚4149 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7697936762880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469731598443008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138491419 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472Ó€)í2Á- _wandbruntime‚3493  _runtime‚3493UÚµà\ÂÖ /loss/FRt3/detection_loss‚2.6852554206190513 2loss/SRqqt2/assignment_loss‚1.6962590997572988 1loss/SRqqt1/detection_loss‚0.8448447621874995 1loss/SRqqt4/detection_loss‚1.5842137976928994 %loss/total_loss‚43.27117117172434 2loss/SRqqt1/assignment_loss‚1.0990138873216007 0loss/FBt1/assignment_loss‚0.0899032412025295 .loss/FBt2/detection_loss‚2.725138768553734 /loss/FBt3/detection_loss‚4.4956634640693665 /loss/FRt2/assignment_loss‚3.031509914818932 2loss/SRqqt4/assignment_loss‚1.7321815710100863 1loss/FBt2/assignment_loss‚0.11898174445377663 $ _timestamp‚1.788441053201866e+09 0loss/FRt4/assignment_loss‚2.7027046040252403 1loss/SRqqt3/assignment_loss‚1.730457447949917 0loss/FBt3/assignment_loss‚0.3700573897610108 0loss/SRqqt2/detection_loss‚1.813940896652639 1loss/SRqqt3/detection_loss‚2.3539641249564385 /loss/FBt1/detection_loss‚0.4189935889672724 0loss/FRt3/assignment_loss‚3.2110028102480133 1loss/FBt4/assignment_loss‚0.14519595447927713 /loss/FRt1/detection_loss‚1.7334648278852305 trainer/global_step‚4149 _runtime‚3493.246462333 _step‚340 0loss/FRt1/assignment_loss‚1.3271788209676743 /loss/FRt2/detection_loss‚2.7722387156065773 epoch‚3 /loss/FRt4/detection_loss‚2.2054795413105577 /loss/FBt4/detection_loss‚2.3835267772277198w^YÉaÃÛ /loss/FRt4/detection_loss‚2.2054795413105577 /loss/FBt4/detection_loss‚2.3835267772277198 /loss/FRt3/detection_loss‚2.6852554206190513 2loss/SRqqt2/assignment_loss‚1.6962590997572988 1loss/SRqqt1/detection_loss‚0.8448447621874995 1loss/SRqqt4/detection_loss‚1.5842137976928994 %loss/total_loss‚43.27117117172434 2loss/SRqqt1/assignment_loss‚1.0990138873216007 0loss/FBt1/assignment_loss‚0.0899032412025295 .loss/FBt2/detection_loss‚2.725138768553734 /loss/FBt3/detection_loss‚4.4956634640693665 /loss/FRt2/assignment_loss‚3.031509914818932 2loss/SRqqt4/assignment_loss‚1.7321815710100863 1loss/FBt2/assignment_loss‚0.11898174445377663 $ _timestamp‚1.788441053201866e+09 0loss/FRt4/assignment_loss‚2.7027046040252403 1loss/SRqqt3/assignment_loss‚1.730457447949917 0loss/FBt3/assignment_loss‚0.3700573897610108 0loss/SRqqt2/detection_loss‚1.813940896652639 1loss/SRqqt3/detection_loss‚2.3539641249564385 /loss/FBt1/detection_loss‚0.4189935889672724 0loss/FRt3/assignment_loss‚3.2110028102480133 1loss/FBt4/assignment_loss‚0.14519595447927713 /loss/FRt1/detection_loss‚1.7334648278852305 trainer/global_step‚4149 _runtime‚3493.246462333 _step‚340 0loss/FRt1/assignment_loss‚1.3271788209676743 /loss/FRt2/detection_loss‚2.7722387156065773 epoch‚3ÔK\m0žÄ‚(:“ ÞãåÔ³ž¯É gpu.0.gpu‚73.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.207' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.441399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓkŸ>/Å‚(:¤ ÞãåÔ÷“îË# proc.memory.rssMB‚ 2628.19921875! disk.rbd2.out‚226.48974609375 network.recv‚ 1.263793e+06 disk.nvme0n1.out‚ 33760.875 disk.dm-0.in‚ 1724.62890625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 1.533203125 cpu‚25.299949130960808# disk.nvme0c0n1.in‚ 1724.62890625 disk.dm-1.in‚0.5625 disk.sda1.in‚0$ memory_percent‚87.57045269012451) proc.memory.availableMB‚ 2036.45703125% disk./.usageGB‚1108.6043663024902 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 9.405986e+06! disk.nvme0n1.in‚ 1724.62890625 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 138.37109375 disk.nvme0c0n1.out‚ 33760.875 disk.dm-0.out‚ 33760.875 disk.drbd1777.out‚0 disk.dm-1.out‚ 138.37109375 disk.sda.in‚0.5625 disk.sda2.in‚0* proc.memory.percent‚16.041254997253418 proc.cpu.threads‚17) disk./.usagePercent‚74.39936920357981 disk.sda.out‚ 138.37109375*“J'’Æ‚(:‡ íãåÔŠÆõÈ gpu.0.gpu‚93.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚151.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.345 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Üäv5Ç‚(:ª íãåÔíþ·Ë disk.rbd1.in‚0! disk.nvme0n1.in‚ 1724.62890625 disk.dm-0.in‚ 1724.62890625$ memory_percent‚87.57214546203613 disk.sda1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚33858.4599609375$ disk./.usageGB‚1108.604347229004 disk.sda3.in‚0.5625 disk.rbd0.in‚0" disk.dm-0.out‚33858.4599609375 disk.dm-1.out‚138.8125 disk.drbd1777.in‚0* proc.memory.percent‚16.041254997253418 cpu‚25.298824780205997 disk.rbd1.out‚ 1.619140625 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.sda3.out‚138.8125' disk.nvme0c0n1.out‚33858.4599609375 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.rbd2.out‚227.68896484375 disk.sda.out‚138.8125 disk.sda2.in‚0 disk.rbd0.out‚0 network.sent‚ 9.514742e+06 disk.sda1.out‚0( proc.memory.availableMB‚ 2036.1796875 network.recv‚ 1.26978e+06 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1724.62890625 disk.dm-1.in‚0.5625# proc.memory.rssMB‚ 2628.19921875) disk./.usagePercent‚74.39936792354214€%ß#žÈ‚(:“ üãåÔ…æéÈ gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚250.601' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.120200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06jØì?É‚(:´ üãåÔ½”Ì" disk.dm-0.out‚33858.7880859375 network.recv‚ 1.274271e+06) disk./.usagePercent‚74.39936792354214 disk.sda1.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.62890625% disk.nvme0n1.out‚33858.7880859375 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1724.62890625 disk.rbd0.in‚0 disk.dm-1.out‚ 139.3046875) proc.memory.availableMB‚ 2036.16796875 cpu‚25.29776397728564 disk.rbd1.in‚0! disk.rbd2.out‚227.72021484375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0$ disk./.usageGB‚1108.604347229004 disk.sda.in‚0.5625 disk.sda3.out‚ 139.3046875 disk.drbd1777.out‚0 network.sent‚ 9.520602e+06 disk.rbd1.out‚ 1.619140625$ memory_percent‚87.57221698760986* proc.memory.percent‚16.041254997253418 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2628.19921875 proc.cpu.threads‚17 disk.sda.out‚ 139.3046875' disk.nvme0c0n1.out‚33858.7880859375 disk.dm-0.in‚ 1724.62890625LÖÀÒ2Ê- _wandbruntime‚3532  _runtime‚3532 íI ÉËà _step‚341 ' lr-AdamW/pg1‚0.00031968893142928704 ' lr-AdamW/pg2‚0.00031968893142928704 trainer/global_step‚4199 % _timestamp‚1.7884410923503027e+09 _runtime‚3532.394983266IŽPÎÌÈ ' lr-AdamW/pg1‚0.00031968893142928704 ' lr-AdamW/pg2‚0.00031968893142928704 trainer/global_step‚4199 % _timestamp‚1.7884410923503027e+09 _runtime‚3532.394983266 _step‚341Õ·ªÈ2Í- _wandbruntime‚3532  _runtime‚3532‰å(ÎúO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467322707241984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78228084 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410482919616512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚4199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78228084 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410482842988544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7787735269888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7772237016986 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467322680850432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40124337 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467322680850432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61857717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475107815910912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459896824351744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚475110442511872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7787735269888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61857723 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467669035940762 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399799483229184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _step‚342 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _runtime‚3532.396085854tZJq(ÏÿO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467322680850432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61857717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475107815910912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459896824351744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚475110442511872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7787735269888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61857723 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467669035940762 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399799483229184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 _step‚342 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _runtime‚3532.396085854 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467322707241984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78228084 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410482919616512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚4199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78228084 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410482842988544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7787735269888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7772237016986 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467322680850432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40124337 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668Ö˜49<2Ð- _wandbruntime‚3533  _runtime‚3533_‚`¨ÑƒN NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61871953 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29161056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7774066359654 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140118271 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69294172 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399891843148288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10685941344768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467779830170546 _runtime‚ 3533.14674366 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69294589 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78246318 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚414 @:DeviceStatsMonitor.ºÁì/êon_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7786967389184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475220214559232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 % _timestamp‚1.7884410931012812e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467430620555776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140118271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40133533 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚343 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7789567611904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78266378 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61871959 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61871953 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7786967389184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475220214559232 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410577738109952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 204244652333î 'ÒˆN S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475217587944960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467430646947328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7786967389184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475220214559232 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410577738109952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61871953 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29161056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7774066359654 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140118271 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69294172 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399891843148288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10685941344768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467779830170546 _runtime‚ 3533.14674366 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69294589 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78246318 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7786967389184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475220214559232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 % _timestamp‚1.7884410931012812e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467430620555776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140118271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40133533 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚343 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7789567611904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78266378 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61871959 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61871953 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480596432027136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚404418885904384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚472719692509184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79143136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7879366118912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70073157 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62569517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29489023 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚465213857637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141712653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7863717741594 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404418857769984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚26470(áÿO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480596432027136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7876765909504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70073660 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚473077575378970 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚472719666117632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10808732838400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚503 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141712653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7863717741594 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404418857769984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79163170 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415227618742784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473080199091606 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62569523 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29489523 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚3572.054499773 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62569517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480599058628096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 % _timestamp‚1.7884411320095935e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480596432027136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚404418885904384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚472719692509184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79143136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7879366118912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70073157 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62569517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29489023 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚465213857637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7881198460928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79181430 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚404511245823488 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29496088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚404511273957888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62583759 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70088966 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141745123 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7868145369138 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7881198460928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480708830675456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚347 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚472827632214528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473185745895910 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79161370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚472827605822976 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7878598238208 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚472827632214528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62583753 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79181430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480708830675456 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480706204061184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141745123 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚465320198811648 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7865547084262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415322513957376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79161370L½únÀ"ä‰N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79161370 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10811189422080 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7881198460928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79181430 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚404511245823488 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29496088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚404511273957888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62583759 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70088966 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141745123 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7868145369138 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7881198460928 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480708830675456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚347 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚472827632214528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚473185745895910 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79161370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚472827605822976 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7878598238208 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚472827632214528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62583753 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79181430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480708830675456 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480706204061184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141745123 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚465320198811648 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7865547084262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚415322513957376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400Û­Öõ2å- _wandbruntime‚3572  _runtime‚3572|KUYæÓ 0loss/FRt3/assignment_loss‚3.3830021884706287 0loss/SRqqt4/detection_loss‚2.008354727178812 1loss/FBt1/assignment_loss‚0.13304997709790253 0loss/FBt4/assignment_loss‚0.1714266554142038 /loss/FRt4/detection_loss‚2.4658460883384055 0loss/SRqqt2/detection_loss‚2.150507976842481 /loss/FBt2/detection_loss‚2.7125507990519204 /loss/FBt3/detection_loss‚2.1199363708496093 epoch‚3 0loss/FBt3/assignment_loss‚0.5170389515347779 /loss/FRt2/detection_loss‚2.5006325735551553 1loss/SRqqt3/detection_loss‚2.3429131507873535 /loss/FBt1/detection_loss‚0.7596942822014712 /loss/FBt4/detection_loss‚2.7500981092453003 /loss/FRt4/assignment_loss‚3.507491165019096 2loss/SRqqt4/assignment_loss‚1.6108785670561094 1loss/SRqqt1/detection_loss‚0.9555588089212587 trainer/global_step‚4249 $ _timestamp‚1.788441132771648e+09 _runtime‚3572.816238553 /loss/FRt1/detection_loss‚2.0286304547140994 %loss/total_loss‚43.95246554479521 _step‚348 0loss/FRt1/assignment_loss‚1.3359448611736298 2loss/SRqqt1/assignment_loss‚1.0779616248711377 2loss/SRqqt2/assignment_loss‚1.3018235803560116 /loss/FRt2/assignment_loss‚3.364343192519211 0loss/SRqqt3/assignment_loss‚1.71433796589413 0loss/FBt2/assignment_loss‚0.4576600445434451 /loss/FRt3/detection_loss‚2.5827834291590586£ô^çØ 1loss/SRqqt3/detection_loss‚2.3429131507873535 /loss/FBt1/detection_loss‚0.7596942822014712 /loss/FBt4/detection_loss‚2.7500981092453003 /loss/FRt4/assignment_loss‚3.507491165019096 2loss/SRqqt4/assignment_loss‚1.6108785670561094 1loss/SRqqt1/detection_loss‚0.9555588089212587 trainer/global_step‚4249 $ _timestamp‚1.788441132771648e+09 _runtime‚3572.816238553 /loss/FRt1/detection_loss‚2.0286304547140994 %loss/total_loss‚43.95246554479521 _step‚348 0loss/FRt1/assignment_loss‚1.3359448611736298 2loss/SRqqt1/assignment_loss‚1.0779616248711377 2loss/SRqqt2/assignment_loss‚1.3018235803560116 /loss/FRt2/assignment_loss‚3.364343192519211 0loss/SRqqt3/assignment_loss‚1.71433796589413 0loss/FBt2/assignment_loss‚0.4576600445434451 /loss/FRt3/detection_loss‚2.5827834291590586 0loss/FRt3/assignment_loss‚3.3830021884706287 0loss/SRqqt4/detection_loss‚2.008354727178812 1loss/FBt1/assignment_loss‚0.13304997709790253 0loss/FBt4/assignment_loss‚0.1714266554142038 /loss/FRt4/detection_loss‚2.4658460883384055 0loss/SRqqt2/detection_loss‚2.150507976842481 /loss/FBt2/detection_loss‚2.7125507990519204 /loss/FBt3/detection_loss‚2.1199363708496093 epoch‚3 0loss/FBt3/assignment_loss‚0.5170389515347779 /loss/FRt2/detection_loss‚2.5006325735551553Üï:0“è‚(:ˆ ¸äåÔÂ’µÉ gpu.0.gpu‚86.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚280.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÚBé‚(:· ¸äåÔéò×Ë& disk.nvme0n1.out‚34055.47021484375 disk.sda.out‚ 141.2265625 disk.sda1.in‚0 disk.dm-0.in‚ 1724.62890625 proc.cpu.threads‚17) disk./.usagePercent‚74.39938840414489 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0# memory_percent‚87.5730037689209( disk.nvme0c0n1.out‚34055.47021484375% disk./.usageGB‚1108.6046524047852! disk.rbd2.out‚231.14599609375 disk.sda2.in‚0! disk.nvme0n1.in‚ 1724.62890625 disk.sda3.out‚ 141.2265625 cpu‚25.29372358981899 disk.rbd1.in‚0 disk.dm-1.out‚ 141.2265625# proc.memory.rssMB‚ 2629.37109375* proc.memory.percent‚16.048407554626465# disk.dm-0.out‚34055.47021484375 network.sent‚ 9.650733e+06 network.recv‚ 1.300549e+06 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.in‚0.5625# disk.nvme0c0n1.in‚ 1724.62890625 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625( proc.memory.availableMB‚ 2036.0390625 disk.rbd1.out‚ 1.7138671875¢~»Ú“ê‚(:ˆ ÇäåÔÐËÈ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚44.0 gpu.0.powerWatts‚136.407' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.2814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Si­8ë‚(:­ ÇäåÔžÃÀË disk.rbd1.out‚ 1.77685546875 disk.rbd2.out‚ 232.310546875 disk.rbd3.in‚0 cpu‚25.292718407872748% disk./.usageGB‚1108.6046524047852 network.sent‚ 9.760028e+06 disk.rbd1.in‚0& disk.nvme0n1.out‚34056.06005859375 disk.sda3.out‚ 141.859375 disk.dm-0.in‚ 1724.62890625) disk./.usagePercent‚74.39938840414489 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34056.06005859375$ memory_percent‚87.57553100585938* proc.memory.percent‚16.054916381835938 network.recv‚ 1.307546e+06 disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1724.62890625 disk.rbd0.in‚0# disk.dm-0.out‚34056.06005859375 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2630.4375 disk.sda.out‚ 141.859375 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1724.62890625 disk.dm-1.out‚ 141.859375$ proc.memory.availableMB‚2035.625ùö0@2ì- _wandbruntime‚3612  _runtime‚3612šÜèœÇíÁ _runtime‚3612.005102026 _step‚349 & lr-AdamW/pg1‚0.0003153855553775243 & lr-AdamW/pg2‚0.0003153855553775243 trainer/global_step‚4299 % _timestamp‚1.7884411719604387e+09ß7óÌîÆ & lr-AdamW/pg1‚0.0003153855553775243 & lr-AdamW/pg2‚0.0003153855553775243 trainer/global_step‚4299 % _timestamp‚1.7884411719604387e+09 _runtime‚3612.005102026 _step‚349Ý© 2ï- _wandbruntime‚3612  _runtime‚3612_‡1ž(ðúO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80058188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41043934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 % _timestamp‚1.7884411719612253e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚470530916350976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29824245 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚3612.006137828 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478116651384832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63281317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41043937 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63281323 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚486087674744320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚486085048143360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚486085048143360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63281317 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70868651 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419972454715392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 trainer/global_step‚4299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7970996967936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63281323 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478488738529814 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478116677776384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7968396758528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29824714 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚470530941778944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70868179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409038288579584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409038260445184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478486114817178 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7968396758528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0tÏ<–(ñÿO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80078222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10934115545088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7955198466202 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80078222 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚486085048143360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63281317 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70868651 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419972454715392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 trainer/global_step‚4299 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7970996967936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63281323 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478488738529814 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478116677776384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7968396758528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29824714 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚470530941778944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70868179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409038288579584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409038260445184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478486114817178 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7968396758528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80058188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41043934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 % _timestamp‚1.7884411719612253e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚470530916350976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29824245 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚3612.006137828 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478116651384832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63281317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41043937 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63281323 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚486087674744320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚486085048143360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478594285334118 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚470637282953216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 % _timestamp‚1.7884411727123177e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7970229087232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚486194820177408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41053130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80076422 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41053133 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7970229087232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63295559 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7972829309952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80096482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409130648498688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63295553 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3612.757513409 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚420067362075648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478224617481728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409130676633088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29831459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7957027808870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80096482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143392041 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80076422 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478596909046962 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70884139 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143371975 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚420067283363840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7972829309952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 trainer/global_step‚4299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478224591090176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478224591090176 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41053133 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7970229087232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63295559 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7972829309952 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80096482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409130648498688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63295553 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3612.757513409 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚420067362075648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478224617481728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409130676633088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29831459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7957027808870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80096482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143392041 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80076422 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478596909046962 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70884139 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143371975 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚420067283363840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7972829309952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 trainer/global_step‚4299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478224591090176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478224591090176 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478594285334118 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚470637282953216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 % _timestamp‚1.7884411727123177e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7970229087232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚486194820177408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41053130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472߉h0ö2õ- _wandbruntime‚3612  _runtime‚3612CÓ…WöÑ 2loss/SRqqt1/assignment_loss‚1.0264244609180901 /loss/FBt1/detection_loss‚0.5338456834731407 0loss/SRqqt3/assignment_loss‚1.83158586536135 .loss/FRt4/detection_loss‚2.479731645694999 1loss/SRqqt1/detection_loss‚1.2269810579590879 /loss/FBt2/detection_loss‚2.1672226020268033 trainer/global_step‚4299 1loss/SRqqt2/detection_loss‚1.7016142420470715 .loss/FBt3/detection_loss‚4.179068922996521 /loss/FRt3/assignment_loss‚4.169516148238347 1loss/SRqqt2/assignment_loss‚1.363423694546024 /loss/FRt3/detection_loss‚2.5356760189669916 _runtime‚3612.758662284 _step‚352 1loss/FBt2/assignment_loss‚0.47180315546159235 1loss/SRqqt3/detection_loss‚2.3835262630667007 % _timestamp‚1.7884411727140753e+09 .loss/FRt2/detection_loss‚2.411156941291898 /loss/FBt4/detection_loss‚2.5375018375260487 1loss/SRqqt4/detection_loss‚1.8692795592546463 &loss/total_loss‚46.050910049766415 epoch‚3 /loss/FRt1/assignment_loss‚1.942728789229142 /loss/FRt2/assignment_loss‚3.108027311258538 1loss/FBt1/assignment_loss‚0.11909846569783908 0loss/FBt3/assignment_loss‚1.0530094305674236 0loss/FBt4/assignment_loss‚0.2229894875947918 .loss/FRt4/assignment_loss‚3.32197319490965 2loss/SRqqt4/assignment_loss‚1.6536101217195391 /loss/FRt1/detection_loss‚1.7411151499601834’îÃ\÷Ö 1loss/SRqqt3/detection_loss‚2.3835262630667007 % _timestamp‚1.7884411727140753e+09 .loss/FRt2/detection_loss‚2.411156941291898 /loss/FBt4/detection_loss‚2.5375018375260487 1loss/SRqqt4/detection_loss‚1.8692795592546463 &loss/total_loss‚46.050910049766415 epoch‚3 /loss/FRt1/assignment_loss‚1.942728789229142 /loss/FRt2/assignment_loss‚3.108027311258538 1loss/FBt1/assignment_loss‚0.11909846569783908 0loss/FBt3/assignment_loss‚1.0530094305674236 0loss/FBt4/assignment_loss‚0.2229894875947918 .loss/FRt4/assignment_loss‚3.32197319490965 2loss/SRqqt4/assignment_loss‚1.6536101217195391 /loss/FRt1/detection_loss‚1.7411151499601834 2loss/SRqqt1/assignment_loss‚1.0264244609180901 /loss/FBt1/detection_loss‚0.5338456834731407 0loss/SRqqt3/assignment_loss‚1.83158586536135 .loss/FRt4/detection_loss‚2.479731645694999 1loss/SRqqt1/detection_loss‚1.2269810579590879 /loss/FBt2/detection_loss‚2.1672226020268033 trainer/global_step‚4299 1loss/SRqqt2/detection_loss‚1.7016142420470715 .loss/FBt3/detection_loss‚4.179068922996521 /loss/FRt3/assignment_loss‚4.169516148238347 1loss/SRqqt2/assignment_loss‚1.363423694546024 /loss/FRt3/detection_loss‚2.5356760189669916 _runtime‚3612.758662284 _step‚352 1loss/FBt2/assignment_loss‚0.47180315546159235àE¾ø‚(:’ ÖäåÔ뺭É gpu.0.gpu‚80.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.119' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.82380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¦˜@ù‚(:µ ÖäåÔîïÒË cpu‚25.291716418821956 proc.cpu.threads‚17% disk./.usageGB‚1108.6046524047852 disk.rbd3.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1724.62890625' disk.nvme0c0n1.out‚34153.5693359375 disk.drbd1777.in‚0) proc.memory.availableMB‚ 2035.89453125 disk.rbd0.in‚0" disk.dm-0.out‚34153.5693359375 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚34153.5693359375 disk.sda3.in‚0.5625 network.recv‚ 1.311315e+06) disk./.usagePercent‚74.39938840414489! disk.nvme0n1.in‚ 1724.62890625 disk.sda.out‚ 142.25390625 disk.dm-1.in‚0.5625 disk.rbd2.in‚0$ memory_percent‚87.57388591766357* proc.memory.percent‚16.054916381835938 network.sent‚ 9.765244e+06 disk.dm-0.in‚ 1724.62890625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.77685546875 disk.rbd2.out‚ 233.353515625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 142.25390625 disk.sda2.in‚0 disk.dm-1.out‚ 142.25390625 disk.drbd1777.out‚0 disk.sda.in‚0.5625 proc.memory.rssMB‚ 2630.4375”„ú‚(:’ åäåÔ½§öÈ gpu.0.gpu‚94.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.50499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00]÷j?û‚(:´ åäåÔú‚ÛË# disk.dm-0.out‚34154.12060546875 disk.drbd1777.in‚0 disk.rbd2.out‚ 233.494140625 disk.sda2.in‚0 network.recv‚ 1.318095e+06 disk.drbd1777.out‚0& disk.nvme0n1.out‚34154.12060546875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34154.12060546875$ memory_percent‚87.57328987121582( proc.memory.availableMB‚ 2035.9921875 cpu‚25.290724263924755 network.sent‚ 9.874376e+06) disk./.usagePercent‚74.39938840414489% disk./.usageGB‚1108.6046524047852 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 142.85546875 disk.sda2.out‚0 disk.dm-1.out‚ 142.85546875 proc.memory.rssMB‚ 2630.4375 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1724.62890625 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1724.62890625 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 142.85546875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-0.in‚ 1724.62890625* proc.memory.percent‚16.054916381835938 disk.sda.in‚0.5625õŠ¸¾’ü‚(:‡ ôäåÔÀí£É gpu.0.gpu‚73.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚42.0 gpu.0.powerWatts‚91.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²®•>ý‚(:³ ôäåÔú¨Ë disk.rbd0.out‚0 disk.dm-1.out‚ 143.17578125 proc.cpu.threads‚17 network.sent‚ 9.884068e+06 disk.rbd2.out‚ 234.537109375( proc.memory.availableMB‚ 2035.8359375 cpu‚25.28974632489423 network.recv‚ 1.330108e+06 disk.sda.out‚ 143.17578125 disk.sda3.out‚ 143.17578125 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.in‚0$ memory_percent‚87.57424354553223 proc.memory.rssMB‚ 2630.4375 disk.rbd1.out‚ 1.83984375 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1724.62890625* proc.memory.percent‚16.054916381835938 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0# disk.dm-0.out‚34154.42919921875) disk./.usagePercent‚74.39938840414489 disk.rbd1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1724.62890625& disk.nvme0n1.out‚34154.42919921875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1724.62890625( disk.nvme0c0n1.out‚34154.42919921875% disk./.usageGB‚1108.6046524047852 disk.sda.in‚0.5625\³Ûk2þ- _wandbruntime‚3652  _runtime‚3652•y¾¡Éÿà _step‚353 ' lr-AdamW/pg1‚0.00031319562100502274 ' lr-AdamW/pg2‚0.00031319562100502274 trainer/global_step‚4349 % _timestamp‚1.7884412119815092e+09 _runtime‚3652.026177395Â5MõÎ€È ' lr-AdamW/pg1‚0.00031319562100502274 ' lr-AdamW/pg2‚0.00031319562100502274 trainer/global_step‚4349 % _timestamp‚1.7884412119815092e+09 _runtime‚3652.026177395 _step‚353á,3Yë2- _wandbruntime‚3652  _runtime‚3652ªÿÙ'˜%‚úO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚475848000492544 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30160876 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3652.027223437 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63993123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71664115 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚475847975064576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41503737 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8060027607552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71664613 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8062627816960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491573664259584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80993274 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80973240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 % _timestamp‚1.7884412119823282e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80973240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8062627816960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80993274 _step‚354 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚483897277968022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41503734 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8046679190810 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491573664259584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚483513636652032 trainer/global_step‚4349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚495 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074{‹¤ô(ƒÿO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413657663120384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚483513663043584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491576290860544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30160381 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424716977072128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8060027607552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144986397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚475848000492544 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30160876 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3652.027223437 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63993123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71664115 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚475847975064576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41503737 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8060027607552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71664613 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8062627816960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491573664259584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80993274 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80973240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 % _timestamp‚1.7884412119823282e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80973240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8062627816960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80993274 _step‚354 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚483897277968022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41503734 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8046679190810 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491573664259584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚483513636652032 trainer/global_step‚4349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚495 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64007359 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4349 _step‚355 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484002824772326 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30167577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424811864097280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11061812923392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491686062907904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71680049 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81011534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64007353 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8061859936256 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41512930 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484005448485170 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475954316238848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64007353 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 _runtime‚3652.784289511 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30167119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80991474 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41512933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Öâ06'†‰N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81011534 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64007359 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4349 _step‚355 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484002824772326 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30167577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424811864097280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11061812923392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491686062907904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71680049 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81011534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64007353 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8061859936256 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41512930 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484005448485170 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475954316238848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64007353 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 _runtime‚3652.784289511 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30167119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80991474 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41512933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚483621576357376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413750079308288ãÄÝk~2‡- _wandbruntime‚3652  _runtime‚3652ˆDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488910648310784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8154258665984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8138159915418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64704917 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64704917 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41963534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11184366464512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30494532 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚429461381669376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489303193693594 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418277093929984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚481165033778176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚358 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8151658456576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41963537 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚481165059206144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81888292 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30494990 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11184315873792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72458066 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8140758200086 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488910621919232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497064906976768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81908326 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489305817406230 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497062280375808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488910621919232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497062280375808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096Ùÿ)(•ÿO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72458066 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8140758200086 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488910621919232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497064906976768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81908326 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489305817406230 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497062280375808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488910621919232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497062280375808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64704923 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8154258665984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146613249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488910648310784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8154258665984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8138159915418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64704917 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64704917 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41963534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11184366464512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30494532 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚429461381669376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489303193693594 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418277093929984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚481165033778176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚358 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8151658456576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41963537 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚481165059206144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81888292 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30494990 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11184315873792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8139989258086 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚489018588016128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8153490785280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64719153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41972733 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚481271400380416 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64719159 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429556352322560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418369453849088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81926586 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72473848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72474305 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30501572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚359 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146645745 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489413987923378 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.cÁ}†âurrent‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8156091008000 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 $ _timestamp‚1.788441252044757e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8153490785280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429556273610752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72473848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72474305 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30501572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚359 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146645745 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489413987923378 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8156091008000 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 $ _timestamp‚1.788441252044757e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8153490785280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429556273610752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8139989258086 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚489018588016128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8153490785280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64719153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41972733 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚481271400380416 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64719159 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429556352322560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418369453849088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81926586 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11309748793856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82823378 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42423337 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65416717 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73253412 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434206293892608 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30830075 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11309797287424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚486482092491776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 % _timestamp‚1.7884412902713194e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚3730.316206914 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494711733131802 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65416723 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82803344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8229640640026 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502550896492032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8245889515008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚486482117919744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8243289305600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚362®Z …ý¥ÿO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494714356844438 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73252936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42423334 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65416717 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30829602 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8232238924694 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494307607186432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8245889515008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚434206217264640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 VBDeviceStatsMonitor.on_train_batch_start/active€B_bytes.all.allocated‚502553523092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494307633577984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82823378 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494307607186432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502553523092992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65416723 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502550896492032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8245889515008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚486482117919744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8243289305600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8243289305600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2647 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚494307633577984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502550896492032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11309748793856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82823378 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42423337 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65416717 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73253412 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434206293892608 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30830075 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11309797287424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚486482092491776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 % _timestamp‚1.7884412902713194e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚3730.316206914 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494711733131802 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65416723 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82803344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8229640640026 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494415546891776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502663295140352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42432530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11312267098112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42432533 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494822527361586 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73268732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _step‚363 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65430959 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 _runtime‚3731.057459866 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65430953 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502663295140352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8247721857024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8234068267570 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚422988856524288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82841638 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494415573283328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73269194 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8245121634304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11312315578368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82821578 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434301200237056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074\Œl'¨‰N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8234068267570 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚422988856524288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82841638 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494415573283328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73269194 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8245121634304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11312315578368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82821578 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434301200237056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82841638 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247721857024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148252531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚434301123622400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494415546891776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502663295140352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42432530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11312267098112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42432533 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494822527361586 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73268732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _step‚363 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65430959 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 _runtime‚3731.057459866 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65430953 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502663295140352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8247721857024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609ëNP 2©- _wandbruntime‚3731  _runtime‚37310ObµYªÓ /loss/FBt2/detection_loss‚2.5491205964769637 _runtime‚3731.058837602 0loss/FBt4/assignment_loss‚0.1415410919871647 0loss/FRt2/assignment_loss‚3.0400401755820874 0loss/FBt1/assignment_loss‚0.1260053884930071 0loss/SRqqt2/detection_loss‚2.018147852109826 epoch‚3 _step‚364 1loss/SRqqt3/detection_loss‚2.5399233885109425 /loss/FBt1/detection_loss‚0.7985548745453744 $ _timestamp‚1.788441291014156e+09 /loss/FRt2/detection_loss‚2.4012892124957816 /loss/FRt1/assignment_loss‚1.405913078226149 2loss/SRqqt2/assignment_loss‚1.4718133123672528 2loss/SRqqt4/assignment_loss‚1.7396989497205904 1loss/SRqqt1/detection_loss‚1.0096761871129274 0loss/SRqqt3/assignment_loss‚2.19637529598549 2loss/FBt2/assignment_loss‚0.014036895108542271 /loss/FRt4/detection_loss‚2.5667842045062925 /loss/SRqqt4/detection_loss‚1.67986786173236 /loss/FBt3/detection_loss‚2.7144051119685173 trainer/global_step‚4449 .loss/FRt4/assignment_loss‚2.49380299085524 0loss/FBt3/assignment_loss‚0.2935820725688245 &loss/total_loss‚43.374883247729954 2loss/SRqqt1/assignment_loss‚1.0535447605619472 /loss/FRt3/detection_loss‚2.7090681322983334 /loss/FBt4/detection_loss‚3.2522033751010895 /loss/FRt1/detection_loss‚1.3968840801971965 0loss/FRt3/assignment_loss‚3.7626043592180523|”(%^«Ø 0loss/SRqqt2/detection_loss‚2.018147852109826 epoch‚3 _step‚364 1loss/SRqqt3/detection_loss‚2.5399233885109425 /loss/FBt1/detection_loss‚0.7985548745453744 $ _timestamp‚1.788441291014156e+09 /loss/FRt2/detection_loss‚2.4012892124957816 /loss/FRt1/assignment_loss‚1.405913078226149 2loss/SRqqt2/assignment_loss‚1.4718133123672528 2loss/SRqqt4/assignment_loss‚1.7396989497205904 1loss/SRqqt1/detection_loss‚1.0096761871129274 0loss/SRqqt3/assignment_loss‚2.19637529598549 2loss/FBt2/assignment_loss‚0.014036895108542271 /loss/FRt4/detection_loss‚2.5667842045062925 /loss/SRqqt4/detection_loss‚1.67986786173236 /loss/FBt3/detection_loss‚2.7144051119685173 trainer/global_step‚4449 .loss/FRt4/assignment_loss‚2.49380299085524 0loss/FBt3/assignment_loss‚0.2935820725688245 &loss/total_loss‚43.374883247729954 2loss/SRqqt1/assignment_loss‚1.0535447605619472 /loss/FRt3/detection_loss‚2.7090681322983334 /loss/FBt4/detection_loss‚3.2522033751010895 /loss/FRt1/detection_loss‚1.3968840801971965 0loss/FRt3/assignment_loss‚3.7626043592180523 /loss/FBt2/detection_loss‚2.5491205964769637 _runtime‚3731.058837602 0loss/FBt4/assignment_loss‚0.1415410919871647 0loss/FRt2/assignment_loss‚3.0400401755820874 0loss/FBt1/assignment_loss‚0.1260053884930071캥$ ž¬‚(:“ ÎååÔÎä‘É gpu.0.gpu‚68.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚252.978' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.595600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y0„Ï;­‚(:° ÎååÔºãÍË& disk.nvme0c0n1.out‚34448.736328125 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 146.64453125 disk.sda1.out‚0$ disk.nvme0n1.out‚34448.736328125 disk.rbd1.in‚0 network.sent‚ 1.0142365e+07 network.recv‚ 1.394857e+06% disk./.usageGB‚1108.6052055358887! disk.nvme0n1.in‚ 1724.62890625 disk.dm-0.in‚ 1724.62890625 disk.sda1.in‚0! disk.dm-0.out‚34448.736328125* proc.memory.percent‚16.054916381835938 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 146.64453125 disk.rbd2.in‚0( proc.memory.availableMB‚ 2035.6171875# disk.nvme0c0n1.in‚ 1724.62890625# memory_percent‚87.5755786895752 cpu‚25.283940119072696 disk.sda.in‚0.5625 disk.rbd0.in‚0 disk.sda2.in‚0! disk.rbd2.out‚240.31884765625 proc.memory.rssMB‚ 2630.4375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 146.64453125) disk./.usagePercent‚74.39942552523735 disk.rbd3.out‚0 disk.sda3.in‚0.5625ÕD=ä‘®‚(:† ÝååÔñ”É gpu.0.gpu‚84.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ÁY— ¯‚(:± ¡P#U/ÝååÔ’ŒËË disk.dm-1.out‚ 147.30859375 disk.sda1.in‚0 disk.sda3.out‚ 147.30859375 disk.dm-0.in‚ 1754.04296875 proc.memory.rssMB‚ 2630.4375 network.recv‚ 1.415861e+06) disk./.usagePercent‚74.39939710840105% disk.nvme0n1.out‚34449.1767578125 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 147.30859375' disk.nvme0c0n1.out‚34449.1767578125$ memory_percent‚87.57572174072266 disk.sda2.out‚0 disk.rbd3.in‚0* proc.memory.percent‚16.054916381835938 disk.sda2.in‚0 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375& proc.memory.availableMB‚ 2035.59375% disk./.usageGB‚1108.6047821044922 disk.sda.in‚0.5625 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚240.4716796875 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.283011292654518 network.sent‚ 1.0258224e+07 disk.rbd3.out‚0" disk.dm-0.out‚34449.1767578125(¬5í“°‚(:ˆ ìååÔ¹¿É gpu.0.gpu‚70.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚257.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼¥ Ù6±‚(:« ìååÔ–ÕË disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0$ memory_percent‚87.57600784301758' proc.memory.availableMB‚ 2035.546875! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 147.83203125 disk.dm-0.in‚ 1754.04296875 disk.sda.out‚ 147.83203125 proc.memory.rssMB‚ 2630.4375 proc.cpu.threads‚17 disk.dm-0.out‚34546.78515625) disk./.usagePercent‚74.39942987736544 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚34546.78515625# disk.nvme0n1.out‚34546.78515625 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.28208851019269 network.recv‚ 1.420313e+06 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1754.04296875 network.sent‚ 1.0263988e+07 disk.drbd1777.out‚0 disk.rbd2.out‚241.5146484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 147.83203125 disk.rbd1.out‚ 1.83984375 disk.sda.in‚0.5625* proc.memory.percent‚16.054916381835938% disk./.usageGB‚1108.6052703857422 disk.sda2.out‚06\©72²- _wandbruntime‚3770  _runtime‚3770)[±ûȳ _step‚365 ' lr-AdamW/pg1‚0.00030647832432474957 ' lr-AdamW/pg2‚0.00030647832432474957 trainer/global_step‚4499 $ _timestamp‚1.788441329958172e+09 _runtime‚3770.002836741""³Í´Ç trainer/global_step‚4499 $ _timestamp‚1.788441329958172e+09 _runtime‚3770.002836741 _step‚365 ' lr-AdamW/pg1‚0.00030647832432474957 ' lr-AdamW/pg2‚0.00030647832432474957íÍÃ-2µ- _wandbruntime‚3770  _runtime‚3770P…Umþ'¶øO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508039512608256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚491799151205376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508042139209216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8334920154624 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11435310200832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚499704592453632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8323719649302 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149866953 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508042139209216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚438951130756096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83718396 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83738430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508039512608256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74046240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚438951209481216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 trainer/global_step‚4499 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8337520364032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11435259610112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8334920154624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31163589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500120272570010 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66128517 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66128523 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427515899280384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42883134 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚438951130756096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83718396 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83738430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508039512608256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74046240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚438951209481216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 trainer/global_step‚4499 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8337520364032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11435259610112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8334920154624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31163589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500120272570010 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66128517 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66128523 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427515899280384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42883134 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491905492379648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42892330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8336752483328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11437807423488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66142753 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427608287333888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500228443086950 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499812558550528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74062500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74062038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31169708 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚367 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66142759 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427608259199488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚4499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149899449 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66142759 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427608259199488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚4499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149899449 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491905492379648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42892330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8336752483328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11437807423488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66142753 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427608287333888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500228443086950 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚499812558550528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74062500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74062038ï;2»- _wandbruntime‚3770  _runtime‚3770£¢Ø¢[¼Õ 1loss/FBt2/assignment_loss‚0.15266746660280558 0loss/SRqqt3/detection_loss‚2.432008781216361 /loss/FBt3/detection_loss‚2.9079864621162415 1loss/SRqqt1/detection_loss‚1.0259633626350586 1loss/SRqqt2/detection_loss‚1.7387384866413318 /loss/FBt2/detection_loss‚2.0183457798428006 epoch‚3 0loss/FRt2/assignment_loss‚2.8000268055045088 1loss/SRqqt2/assignment_loss‚1.385994724014349 trainer/global_step‚4499 /loss/FBt1/detection_loss‚0.6961624493927393 .loss/FRt2/detection_loss‚2.414709706669268 0loss/FRt1/assignment_loss‚1.9930582446838492 1loss/FBt3/assignment_loss‚0.16441619214601816 /loss/FRt1/detection_loss‚1.3529462676439712 0loss/SRqqt4/detection_loss‚1.750303759894989 _step‚368 0loss/FBt4/assignment_loss‚0.1597986841807142 .loss/FBt4/detection_loss‚2.629666455090046 2loss/SRqqt1/assignment_loss‚1.1491533163934946 2loss/SRqqt4/assignment_loss‚1.5726908780634403 1loss/FBt1/assignment_loss‚0.17151736955380745 /loss/FRt3/detection_loss‚2.5453549515129437 %loss/total_loss‚42.27176086509339 0loss/FRt4/assignment_loss‚3.4555981768214186 % _timestamp‚1.7884413306974354e+09 _runtime‚3770.742010339 /loss/FRt3/assignment_loss‚3.622528957598137 2loss/SRqqt3/assignment_loss‚1.7435688414808475 .loss/FRt4/detection_loss‚2.388554745394251ð,m`½Ú _runtime‚3770.742010339 /loss/FRt3/assignment_loss‚3.622528957598137 2loss/SRqqt3/assignment_loss‚1.7435688414808475 .loss/FRt4/detection_loss‚2.388554745394251 1loss/FBt2/assignment_loss‚0.15266746660280558 0loss/SRqqt3/detection_loss‚2.432008781216361 /loss/FBt3/detection_loss‚2.9079864621162415 1loss/SRqqt1/detection_loss‚1.0259633626350586 1loss/SRqqt2/detection_loss‚1.7387384866413318 /loss/FBt2/detection_loss‚2.0183457798428006 epoch‚3 0loss/FRt2/assignment_loss‚2.8000268055045088 1loss/SRqqt2/assignment_loss‚1.385994724014349 trainer/global_step‚4499 /loss/FBt1/detection_loss‚0.6961624493927393 .loss/FRt2/detection_loss‚2.414709706669268 0loss/FRt1/assignment_loss‚1.9930582446838492 1loss/FBt3/assignment_loss‚0.16441619214601816 /loss/FRt1/detection_loss‚1.3529462676439712 0loss/SRqqt4/detection_loss‚1.750303759894989 _step‚368 0loss/FBt4/assignment_loss‚0.1597986841807142 .loss/FBt4/detection_loss‚2.629666455090046 2loss/SRqqt1/assignment_loss‚1.1491533163934946 2loss/SRqqt4/assignment_loss‚1.5726908780634403 1loss/FBt1/assignment_loss‚0.17151736955380745 /loss/FRt3/detection_loss‚2.5453549515129437 %loss/total_loss‚42.27176086509339 0loss/FRt4/assignment_loss‚3.4555981768214186 % _timestamp‚1.7884413306974354e+09ðv—}_¾‚(:’ ûååÔúî³É gpu.0.gpu‚71.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â ˆ½;¿‚(:° ûååÔÜî³Ë disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 148.34765625 disk.drbd1777.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚34551.431640625 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚34551.431640625 disk.dm-1.out‚ 148.34765625) proc.memory.availableMB‚ 2035.30859375% disk./.usageGB‚1108.6088638305664# disk.nvme0c0n1.in‚ 1754.04296875# memory_percent‚87.5774621963501 disk.sda.in‚0.5625 disk.sda1.in‚0$ disk.nvme0n1.out‚34551.431640625 disk.rbd0.out‚0 proc.memory.rssMB‚ 2630.4375! disk.rbd2.out‚242.67529296875 disk.rbd3.in‚0 network.sent‚ 1.027036e+07 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd1.out‚ 1.83984375 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1754.04296875* proc.memory.percent‚16.054916381835938 cpu‚25.281177559802597 disk.sda1.out‚0 network.recv‚ 1.425171e+06) disk./.usagePercent‚74.39967103646268 disk.rbd1.in‚0 disk.sda.out‚ 148.34765625õD6“À‚(:ˆ ŠæåÔå¦çÈ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚48.0 gpu.0.powerWatts‚251.333' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öki 9Á‚(:® ŠæåÔßÃôË# disk.nvme0c0n1.in‚ 1754.04296875$ memory_percent‚87.57748603820801 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.sda1.out‚0 disk.sda3.out‚ 148.64453125 disk.rbd0.in‚0 disk.dm-1.out‚ 148.64453125! disk.nvme0n1.in‚ 1754.04296875 disk.sda.in‚0.5625 disk.sda.out‚ 148.64453125 disk.dm-0.out‚34648.98828125 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.83984375 cpu‚25.280263470071365% disk./.usageGB‚1108.6088638305664 disk.sda2.out‚0 disk.rbd3.in‚0% disk.nvme0c0n1.out‚34648.98828125 disk.rbd1.in‚0* proc.memory.percent‚16.054916381835938 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0378525e+07# disk.nvme0n1.out‚34648.98828125 disk.rbd0.out‚0( proc.memory.availableMB‚ 2035.3046875 proc.memory.rssMB‚ 2630.4375 network.recv‚ 1.431441e+06) disk./.usagePercent‚74.39967103646268! disk.rbd2.out‚243.71826171875 disk.sda2.in‚0u_t:2Â- _wandbruntime‚3809  _runtime‚3809Î’^ÉÃà trainer/global_step‚4549 % _timestamp‚1.7884413692926667e+09 _runtime‚3809.337350615 _step‚369 ' lr-AdamW/pg1‚0.00030419161633738574 ' lr-AdamW/pg2‚0.00030419161633738574Þ>é ÎÄÈ ' lr-AdamW/pg1‚0.00030419161633738574 ' lr-AdamW/pg2‚0.00030419161633738574 trainer/global_step‚4549 % _timestamp‚1.7884413692926667e+09 _runtime‚3809.337350615 _step‚369ñŽž 2Å- _wandbruntime‚3809  _runtime‚3809'!Õ*ÿ'ÆùO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚505531435720854 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚505528812008218 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31497568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443695877826560 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66840323 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8426551003648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513528128724480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505101577720832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84633448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚497116209918976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚505101577720832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43342934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 $ _timestamp‚1.788441369293408e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚497116235346944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513528128724480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66840317 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚505101604112384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8429151213056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8426551003648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31498046 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66840323 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8412602089242 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ïŸ_„ÇþO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66840323 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8426551003648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513528128724480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505101577720832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84633448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚497116209918976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 25989XE€284668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚505101577720832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43342934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 $ _timestamp‚1.788441369293408e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚497116235346944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513528128724480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66840317 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚505101604112384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8429151213056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8426551003648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31498046 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66840323 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8412602089242 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚505531435720854 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚505528812008218 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31497568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443695877826560 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074òeâ¨U“È‚(:ˆ ™æåÔª´øÉ gpu.0.gpu‚81.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚238.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ΨëAÉ‚(:¶ ™æåÔ‚ÅÔÊ disk.drbd1777.in‚0 disk.sda.out‚ 149.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 149.29296875 disk.dm-0.in‚ 1754.04296875 cpu‚25.27938029763984 disk.rbd1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34649.64501953125 disk.dm-1.out‚ 149.29296875 network.sent‚ 1.0384465e+07 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375! disk.rbd2.out‚244.93310546875 disk.rbd3.out‚0& disk.nvme0n1.out‚34649.64501953125( proc.memory.availableMB‚ 2035.0703125 proc.memory.rssMB‚ 2630.4375% disk./.usageGB‚1108.6088638305664 disk.sda.in‚0.5625# disk.dm-0.out‚34649.64501953125* proc.memory.percent‚16.054916381835938) disk./.usagePercent‚74.39967103646268 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1754.04296875! disk.nvme0n1.in‚ 1754.04296875 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625$ memory_percent‚87.57891654968262 network.recv‚ 1.435894e+06 proc.cpu.threads‚17äÕ2œ2Ê- _wandbruntime‚3810  _runtime‚3810r_[¤ '˃N REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66854559 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8417029716786 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84651682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443790694810624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513637900758528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8430983555072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66854559 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84671742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43352133 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31504621 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74856298 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚505209517426176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8428383332352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11563083513344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚497222576521216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513640527372800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151526301 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84671742 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8430983555072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66854553 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43352130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8428383332352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11563032935936 $ _timestamp‚1.788441370032892e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74856754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432227661874688ê1#'̈N YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8430983555072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66854553 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43352130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8428383332352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11563032935936 $ _timestamp‚1.788441370032892e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74856754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432227661874688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66854559 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8417029716786 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84651682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443790694810624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513637900758528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8430983555072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66854559 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84671742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43352133 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31504621 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2647 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74856298 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚505209517426176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8428383332352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11563083513344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚497222576521216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513640527372800 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151526301 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84671742 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0óX†2Í- _wandbruntime‚3810  _runtime‚3810’b.j^ÎØ 1loss/FBt2/assignment_loss‚0.06212916311342269 /loss/FRt4/detection_loss‚2.7482722045646772 1loss/SRqqt2/detection_loss‚1.6260312305142481 /loss/FBt1/detection_loss‚0.6223059210512373 0loss/FRt2/assignment_loss‚2.8308822701616987 2loss/SRqqt3/assignment_loss‚1.5701575432904065 /loss/FRt1/assignment_loss‚1.171844253540039 2loss/SRqqt1/assignment_loss‚0.8767076639924198 /loss/FBt3/detection_loss‚2.6277118259006076 %loss/total_loss‚42.49531904295292 trainer/global_step‚4549 0loss/FBt3/assignment_loss‚0.5431714950439831 /loss/FRt3/detection_loss‚2.6489278231897853 1loss/SRqqt1/detection_loss‚0.7052076974650845 /loss/FBt4/detection_loss‚2.8990129232406616 .loss/FBt2/detection_loss‚2.715864157676697 0loss/FRt4/assignment_loss‚3.2315557632181378 2loss/SRqqt4/assignment_loss‚1.3667508873135545 3loss/FBt4/assignment_loss‚0.0025397726306234996 /loss/FRt1/detection_loss‚1.4954067476838828 0loss/FRt3/assignment_loss‚3.9727105671359646 0loss/SRqqt4/detection_loss‚2.183520097545413 % _timestamp‚1.7884413700349543e+09 0loss/SRqqt3/detection_loss‚2.576509626582265 epoch‚3 _step‚372 2loss/SRqqt2/assignment_loss‚1.3411590594177445 1loss/FBt1/assignment_loss‚0.12400074398511826 -loss/FRt2/detection_loss‚2.55293960469525 _runtime‚3810.079512316òÅ cÏÝ 0loss/FRt2/assignment_loss‚2.8308822701616987 2loss/SRqqt3/assignment_loss‚1.5701575432904065 /loss/FRt1/assignment_loss‚1.171844253540039 2loss/SRqqt1/assignment_loss‚0.8767076639924198 /loss/FBt3/detection_loss‚2.6277118259006076 %loss/total_loss‚42.49531904295292 trainer/global_step‚4549 0loss/FBt3/assignment_loss‚0.5431714950439831 /loss/FRt3/detection_loss‚2.6489278231897853 1loss/SRqqt1/detection_loss‚0.7052076974650845 /loss/FBt4/detection_loss‚2.8990129232406616 .loss/FBt2/detection_loss‚2.715864157676697 0loss/FRt4/assignment_loss‚3.2315557632181378 2loss/SRqqt4/assignment_loss‚1.3667508873135545 3loss/FBt4/assignment_loss‚0.0025397726306234996 /loss/FRt1/detection_loss‚1.4954067476838828 0loss/FRt3/assignment_loss‚3.9727105671359646 0loss/SRqqt4/detection_loss‚2.183520097545413 % _timestamp‚1.7884413700349543e+09 0loss/SRqqt3/detection_loss‚2.576509626582265 epoch‚3 _step‚372 2loss/SRqqt2/assignment_loss‚1.3411590594177445 1loss/FBt1/assignment_loss‚0.12400074398511826 -loss/FRt2/detection_loss‚2.55293960469525 _runtime‚3810.079512316 1loss/FBt2/assignment_loss‚0.06212916311342269 /loss/FRt4/detection_loss‚2.7482722045646772 1loss/SRqqt2/detection_loss‚1.6260312305142481 /loss/FBt1/detection_loss‚0.6223059210512373ôbÇwê“Ђ(:ˆ ¨æåÔ±ù¬É gpu.0.gpu‚88.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚45.0 gpu.0.powerWatts‚158.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™cÑ)6Ñ‚(:« ¨æåÔ†ÆßË disk.dm-0.in‚ 1754.04296875 disk.rbd0.out‚0 disk.sda.in‚0.5625 disk.rbd3.in‚0 cpu‚25.278472149335094 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375! disk.rbd2.out‚246.00732421875 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.04818e+07) disk./.usagePercent‚74.39967103646268! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 149.62109375# disk.nvme0c0n1.in‚ 1754.04296875" memory_percent‚87.579345703125# disk.dm-0.out‚34649.95751953125 disk.dm-1.in‚0.5625* proc.memory.percent‚16.054916381835938& disk.nvme0n1.out‚34649.95751953125 disk.rbd0.in‚0 disk.dm-1.out‚ 149.62109375 disk.sda1.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2630.4375 network.recv‚ 1.442503e+06 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚34649.95751953125 disk.sda1.out‚0 disk.sda2.out‚0% disk./.usageGB‚1108.6088638305664 disk.sda3.in‚0.5625 disk.sda.out‚ 149.62109375 proc.memory.availableMB‚2035ãªçHÒ‚(:’ ·æåÔ“ÆïÈ gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚259.176' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.83519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"àE;Ó‚(:° ·æåÔ€þË disk.dm-1.out‚ 150.29296875 network.recv‚ 1.448786e+06# disk.nvme0c0n1.in‚ 1754.04296875# disk.dm-0.out‚34747.63427734375 disk.rbd1.in‚0 disk.sda3.in‚0.5625 disk.rbd2.out‚ 248.2734375 disk.sda.out‚ 150.29296875 disk.rbd3.out‚0 proc.memory.rssMB‚ 2630.4375 cpu‚25.277582724497417 disk.rbd0.out‚0( disk.nvme0c0n1.out‚34747.63427734375) disk./.usagePercent‚74.39967206049282 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda3.out‚ 150.29296875# memory_percent‚87.5772476196289 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1754.04296875 disk.rbd2.in‚0 disk.sda.in‚0.5625 proc.cpu.threads‚17* proc.memory.percent‚16.054916381835938 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 network.sent‚ 1.0580219e+07% disk./.usageGB‚1108.6088790893555! disk.nvme0n1.in‚ 1754.04296875& disk.nvme0n1.out‚34747.63427734375& proc.memory.availableMB‚ 2035.34375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0§ S2Ô- _wandbruntime‚3848  _runtime‚3848 ¡*)ÇÕÁ & lr-AdamW/pg1‚0.0003018818939513007 & lr-AdamW/pg2‚0.0003018818939513007 trainer/global_step‚4599 % _timestamp‚1.7884414088876202e+09 _runtime‚3848.932267067 _step‚373g,afÖÆ _runtime‚3848.932267067 _step‚373 & lr-AdamW/pg1‚0.0003018818939513007 & lr-AdamW`—rðf/pg2‚0.0003018818939513007 trainer/global_step‚4599 % _timestamp‚1.7884414088876202e+09õûЕ2×- _wandbruntime‚3848  _runtime‚3848ø’ &(ØúO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510939975159062 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8504082813850 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8520782062080 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8518181852672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519016744840704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448440512370688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510937351446426 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43802737 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31832804 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8506681098518 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519019371441664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11685757149184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519016744840704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67552123 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672¹iîþ(ÙÿO trainer/global_step‚4599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510498562988032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436754676496384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67552117 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85568534 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚502433294060544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43802737 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31832804 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8506681098518 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519019371441664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11685757149184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519016744840704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67552123 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510939975159062 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8504082813850 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8520782062080 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8518181852672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519016744840704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448440512370688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510937351446426 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510606529084928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510606529084928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11688296479232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67566359 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85566734 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚502539635234816 _runtime‚3849.679880712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚502539609806848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8520014181376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31839310 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85586794 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788441409634704e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43811933 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511045521963366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519129143489024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511048145676210 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67566359 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448535310451712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519129143489024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448535389163520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436847092684288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75650782 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚3750YF÷܈N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67566353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67566353 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788441409634704e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43811933 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511045521963366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519129143489024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153133087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 370700288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511048145676210 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67566359 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448535310451712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519129143489024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448535389163520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436847092684288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75650782 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚375 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8522614404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4599 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510606529084928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510606529084928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1411 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11688296479232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67566359 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85566734 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚502539635234816 _runtime‚3849.679880712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3298820096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚502539609806848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8520014181376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31839310 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1573 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85586794 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 Fã‚(:³ ÕæåÔÒÍèË disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 151.3359375 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.sda.out‚ 151.3359375 disk.sda2.in‚0* proc.memory.percent‚16.054940223693848 network.recv‚ 1.458928e+06# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.rbd1.in‚0$ memory_percent‚87.58046627044678% disk./.usageGB‚1108.6086921691895" disk.dm-0.out‚34846.2431640625 disk.sda.in‚0.5625 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 1.83984375 disk.rbd2.out‚ 250.54296875 disk.sda2.out‚0% disk.nvme0n1.out‚34846.2431640625 disk.rbd0.out‚0 network.sent‚ 1.0694521e+07' disk.nvme0c0n1.out‚34846.2431640625 disk.sda3.in‚0.5625 disk.rbd0.in‚0) proc.memory.availableMB‚ 2034.81640625 cpu‚25.275833139953615 disk.dm-1.out‚ 151.3359375 disk.sda1.in‚0# proc.memory.rssMB‚ 2630.44140625) disk./.usagePercent‚74.39965951612363A]ªœä‚(:‘ äæåÔºÚ÷È gpu.0.gpu‚62.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚271.89' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.37799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê´t8å‚(:­ äæåÔç”ÀÌ disk.sda3.in‚0.5625% disk./.usageGB‚1108.6086349487305 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda2.in‚0 disk.rbd0.in‚0 network.recv‚ 1.463513e+06 disk.drbd1777.out‚0 disk.sda.out‚ 151.734375 disk.rbd3.in‚0% disk.nvme0n1.out‚34846.5712890625 disk.dm-1.out‚ 151.734375$ memory_percent‚87.57953643798828& proc.memory.availableMB‚ 2034.96875 disk.sda1.in‚0) disk./.usagePercent‚74.39965567601062 disk.rbd2.in‚0 disk.sda3.out‚ 151.734375' disk.nvme0c0n1.out‚34846.5712890625" disk.dm-0.out‚34846.5712890625 disk.sda.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.in‚0.5625# proc.memory.rssMB‚ 2630.44140625* proc.memory.percent‚16.054940223693848 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 cpu‚25.274957895171358 network.sent‚ 1.0700504e+07 disk.drbd1777.in‚0 disk.rbd2.out‚ 252.69140625 disk.dm-0.in‚ 1754.04296875ïßÂ2æ- _wandbruntime‚3888  _runtime‚38886Ê8ßÇçÁ % _timestamp‚1.7884414488415308e+09 _runtime‚3888.886176135 _step‚377 & lr-AdamW/pg1‚0.0002995496385091779 & lr-AdamW/pg2‚0.0002995496385091779 trainer/global_step‚46496ì\1ÌèÆ & lr-AdamW/pg2‚0.0002995496385091779 trainer/global_step‚4649 % _timestamp‚1.7884414488415308e+09 _runtime‚3888.886176135 _step‚377 & lr-AdamW/pg1‚0.0002995496385091779ù sOÄ2é- _wandbruntime‚3888  _runtime‚3888ýÄÍ(êúO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚524505360956928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76428264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453184762209792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68263923 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68263917 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚515895548255232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86483586 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44262537 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86483586 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515895574646784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86463552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚516348514597270 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884414488424292e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚516345890884634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11810683038208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68263923 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441374079171584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11810733628928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚3888.887434997 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚515895548255232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚507750352774144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚524507987557888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8612412911104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32165727 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8609812701696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚524505360956928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76428264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453184762209792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68263923 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68263917 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚515895548255232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86483586 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44262537 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86483586 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515895574646784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86463552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚516348514597270 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884414488424292e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚516345890884634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1411 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11810683038208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68263923 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2686470144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441374079171584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11810733628928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚3888.887434997 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚515895548255232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚507750352774144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚524507987557888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8612412911104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32165727 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8609812701696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 370700288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453279655957504 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8614245253120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68278153 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44271733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11813239309824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68278159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚524617759605248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚524617759605248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86481786 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788441449594719e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516003514352128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76444127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68278159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚516454061401574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚516456685114418 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441466495359488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _step‚379 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86481786 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76443651 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453279734669312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647,1_ 'î†N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516003487960576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8611645030400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44271730 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8597392881126 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76443651 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453279734669312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68278153 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚524615132990976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453279655957504 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8614245253120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68278153 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44271733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11813239309824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68278159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚524617759605248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚524617759605248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86481786 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788441449594719e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516003514352128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76444127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3298820096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68278159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚516454061401574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚812 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2686470144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚516456685114418 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441466495359488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _step‚379 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1573 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86481786 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68975723 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77221782 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8689642547734 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87398638 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11935553204736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚521754430322842 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8704043760128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77222284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521292559913984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8701443550720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8704043760128 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32499947 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 _runtime‚3928.649754088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚382 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68975723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚521757054035478 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚529993977073152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87378604 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529993977073152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 trainer/global_step‚4699 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445993481846784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884414886048393e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44722334 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ÞßØ(ûÿO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2692760576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87398638 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11935605892608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513067411487744 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44722337 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23668457472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8701443550720 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457929115873792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156354321 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68975717 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚521292533522432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚521292533522432 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3298820096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68975723 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77221782 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3298820096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8689642547734 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23668457472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87398638 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11935553204736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚521754430322842 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8704043760128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77222284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521292559913984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8701443550720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8704043760128 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32499947 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3298820096 _runtime‚3928.649754088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚382 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68975723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚521757054035478 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23668457472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1573 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚529993977073152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87378604 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529993977073152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 trainer/global_step‚4699 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445993481846784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884414886048393e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44722334 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529996603674112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513067386059776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2647 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457929035051520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1573 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529996603674112 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074þMÓ¼2ü- _wandbruntime‚3929  _runtime‚3929ãbU‚ 'ý„N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3929.396781506 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386791 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚521400499619328 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513173727234048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚530106375721472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87396838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44731533 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68989959 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446085869900288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 374775808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8691471890610 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68989953 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8703275879424 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32506543 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚521865224552626 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156406857 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚521400473227776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87416898 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458023913451008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚530106375721472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8703275879424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8688873605734 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77238076 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521400499619328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77237618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11938043550720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚521862600839782 _step‚383 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87416898 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376vîöSþ‰N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3298820096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77238076 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521400499619328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77237618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11938043550720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚521862600839782 _step‚383 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87416898 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3929.396781506 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386791 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚521400499619328 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513173727234048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚530106375721472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23668457472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87396838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44731533 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68989959 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446085869900288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 374775808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8691471890610 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23668457472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1573 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68989953 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8703275879424 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32506543 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2647 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚521865224552626 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156406857 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚521400473227776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87416898 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458023913451008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚530106375721472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8703275879424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23668457472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8688873605734 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272ÿ9¤‘ö2ÿ- _wandbruntime‚3929  _runtime‚3929ýS²FX€Ò 1loss/SRqqt2/detection_loss‚1.7003212985667315 /loss/SRqqt4/detection_loss‚2.00132025431755 /loss/FBt2/detection_loss‚2.4727636250582608 trainer/global_step‚4699 /loss/FRt3/assignment_loss‚4.396930275424834 0loss/FRt4/assignment_loss‚3.1344465123282537 .loss/FRt4/detection_loss‚2.445136131180657 /loss/FRt2/assignment_loss‚2.807658866047859 2loss/SRqqt2/assignment_loss‚1.3849116604436527 0loss/FBt2/assignment_loss‚0.4077651623467153 .loss/FBt3/detection_loss‚2.705163925886154 /loss/FBt4/detection_loss‚3.3582902380398343 &loss/total_loss‚44.984051266170205 1loss/FBt4/assignment_loss‚0.17897637869464234 /loss/FRt1/assignment_loss‚1.383522870405665 1loss/FBt3/assignment_loss‚0.30540619464591146 /loss/FRt1/detection_loss‚1.7114162757149283 0loss/SRqqt1/detection_loss‚1.054872074888812 /loss/FBt1/detection_loss‚0.6138828755958992 /loss/FRt2/detection_loss‚2.6126120896501974 1loss/SRqqt3/detection_loss‚2.3543235371189732 _runtime‚3929.398337248 2loss/SRqqt1/assignment_loss‚1.0852225780900981 epoch‚3 .loss/FRt3/detection_loss‚2.652615770216912 _step‚384 % _timestamp‚1.7884414893537793e+09 1loss/SRqqt3/assignment_loss‚2.232874519401981 2loss/SRqqt4/assignment_loss‚1.6787496961133426 0loss/FBt1/assignment_loss‚0.3048684559923403EÌm]× % _timestamp‚1.7884414893537793e+09 1loss/SRqqt3/assignment_loss‚2.232874519401981 2loss/SRqqt4/assignment_loss‚1.6787496961133426 0loss/FBt1/assignment_loss‚0.3048684559923403 1loss/SRqqt2/detection_loss‚1.7003212985667315 /loss/SRqqt4/detection_loss‚2.00132025431755 /loss/FBt2/detection_loss‚2.4727636250582608 trainer/global_step‚4699 /loss/FRt3/assignment_loss‚4.396930275424834 0loss/FRt4/assignment_loss‚3.1344465123282537 .loss/FRt4/detection_loss‚2.445136131180657 /loss/FRt2/assignment_loss‚2.807658866047859 2loss/SRqqt2/assignment_loss‚1.3849116604436527 0loss/FBt2/assignment_loss‚0.4077651623467153 .loss/FBt3/detection_loss‚2.705163925886154 /loss/FBt4/detection_loss‚3.3582902380398343 &loss/total_loss‚44.984051266170205 1loss/FBt4/assignment_loss‚0.17897637869464234 /loss/FRt1/assignment_loss‚1.383522870405665 1loss/FBt3/assignment_loss‚0.30540619464591146 /loss/FRt1/detection_loss‚1.7114162757149283 0loss/SRqqt1/detection_loss‚1.054872074888812 /loss/FBt1/detection_loss‚0.6138828755958992 /loss/FRt2/detection_loss‚2.6126120896501974 1loss/SRqqt3/detection_loss‚2.3543235371189732 _runtime‚3929.398337248 2loss/SRqqt1/assignment_loss‚1.0852225780900981 epoch‚3 .loss/FRt3/detection_loss‚2.652615770216912 _step‚384€AqùÈž‚‚(:“ ‘çåÔñ»†É gpu.0.gpu‚69.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.068599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ƒ«ÆBƒ‚(:· ‘çåÔÄ Ë) proc.memory.availableMB‚ 2035.15234375! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 153.16796875( disk.nvme0c0n1.out‚34945.29638671875 disk.dm-1.out‚ 153.16796875 disk.sda.out‚ 153.16796875 disk.rbd0.out‚0 network.sent‚ 1.0820768e+07 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625 network.recv‚ 1.478407e+06 disk.rbd0.in‚0# memory_percent‚87.5784158706665* proc.memory.percent‚16.054940223693848% disk./.usageGB‚1108.6086044311523 disk.drbd1777.in‚0 disk.sda1.in‚0 cpu‚25.27241603548723 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1754.04296875 disk.rbd1.out‚ 1.83984375 disk.rbd2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2630.44140625 disk.dm-0.in‚ 1754.04296875# disk.dm-0.out‚34945.29638671875 disk.drbd1777.out‚0 disk.sda.in‚0.5625) disk./.usagePercent‚74.39965362795036 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚34945.29638671875 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd2.out‚ 257.22265625‚(ž„‚(:“  çåÔ—»¢É gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚28.19343566894531* gpu.0.memoryAllocatedBytes‚ 24217976832 gpu.0.temp‚47.0 gpu.0.powerWatts‚239.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.948800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tØøÆC…‚(:¸  çåÔˆ¬¯Ë disk.sda2.in‚0 disk.sda3.out‚ 153.73828125 disk.rbd2.in‚0 network.recv‚ 1.485046e+06 disk.rbd3.in‚0& disk.nvme0n1.out‚35042.80029296875 disk.dm-1.in‚0.5625 disk.rbd2.out‚ 259.37109375 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875# proc.memory.rssMB‚ 2630.44140625) disk./.usagePercent‚74.39964978783735 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda1.in‚0( proc.memory.availableMB‚ 2035.0703125 network.sent‚ 1.0929385e+07 disk.sda3.in‚0.5625# disk.dm-0.out‚35042.80029296875 disk.drbd1777.in‚0% disk./.usageGB‚1108.6085472106934 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.054940223693848 cpu‚25.271637473801174 disk.sda2.out‚0( disk.nvme0c0n1.out‚35042.80029296875 disk.dm-0.in‚ 1754.04296875 disk.drbd1777.out‚0 disk.sda.out‚ 153.73828125 proc.cpu.threads‚17 disk.dm-1.out‚ 153.73828125$ memory_percent‚87.57891654968262GÀ§Ø”†‚(:‰ ¯çåÔôåÊ gpu.0.gpu‚54.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚266.132' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôí$ïA‡‚(:¶ ¯çåÔŸ’õË disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 154.17578125 disk.rbd1.out‚ 1.83984375 disk.sda.in‚0.5625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.054940223693848 cpu‚25.270733634193576 disk.drbd1777.in‚0 disk.rbd2.out‚ 261.60546875 disk.sda1.out‚0 disk.sda3.out‚ 154.17578125% disk./.usageGB‚1108.6085472106934 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.04296875# proc.memory.rssMB‚ 2630.44140625 network.sent‚ 1.0935777e+07 network.recv‚ 1.489536e+06) disk./.usagePercent‚74.39964978783735% disk.nvme0n1.out‚35043.7177734375' disk.nvme0c0n1.out‚35043.7177734375$ memory_percent‚87.58046627044678) proc.memory.availableMB‚ 2034.81640625 disk.sda.out‚ 154.17578125 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875" disk.dm-0.out‚35043.7177734375 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.rbd3.in‚0€ZbÈ2ˆ- _wandbruntime‚3968  _runtime‚3968“gW>ȉ _runtime‚3968.495047441 _step‚385 ' lr-AdamW/pg1‚0.00029481947720553836 ' lr-AdamW/pg2‚0.00029481947720553836 trainer/global_step‚4749 $ _timestamp‚1.788441528450374e+09—²šùÍŠÇ _runtime‚3968.495047441 _step‚385 ' lr-AdamW/pg1‚0.00029481947720553836 ' lr-AdamW/pg2‚0.00029481947720553836 trainer/global_step‚4749 $ _timestamp‚1.788441528450374e+09ô-û‚2‹- _wandbruntime‚3968  _runtime‚3968Ñæÿ'ŒùO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12060086664192 $ _timestamp‚1.788441528451148e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450612912656384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3968.496115613 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88293656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32833232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462672971186176 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88313690 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78015862 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88313690 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8778524987674 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88293656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚535485219790336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚493 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12060137254912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526689518789632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69687523 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚518384470201344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 _step‚386 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69687517 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚535485219790336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8793074399744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462673049911296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450612884521984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69687517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45182134 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚526689545181184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8781123272342 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚535482593189376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526689545181184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8795674609152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78015366 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚527162969761050 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8795674609152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8793074399744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526689518789632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45182137 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚527165593473686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚535482593189376 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672Îm”ñ_%þO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69687517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45182134 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚526689545181184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8781123272342 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚535482593189376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526689545181184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8795674609152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78015366 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚527162969761050 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8795674609152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8793074399744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚526689518789632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45182137 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚527165593473686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚535482593189376 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12060086664192 $ _timestamp‚1.788441528451148e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450612912656384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3968.496115613 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88293656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32833232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462672971186176 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88313690 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78015862 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88313690 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8778524987674 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88293656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚535485219790336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚493 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12060137254912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526689518789632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69687523 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚518384470201344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 _step‚386 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69687517 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚535485219790336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8793074399744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462673049911296 P@DeviceStatsMonitor.on_train_batch_DÊ¥start/requested_bytes.all.peak‚ 20424465233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450612884521984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648‚¸¿0O2Ž- _wandbruntime‚3969  _runtime‚3969gYÆ 'ƒN ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69701759 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8780354330342 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3969.251080758 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8797506951168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚535594991837696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚526797484886528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88311890 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462767858784256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12062586208768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45191333 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462767937496064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78031280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69701753 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚535592365223424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526797458494976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32840407 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78031280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69701753 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚535592365223424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526797458494976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32840407ƒ Ïl2‘- _wandbruntime‚3969  _runtime‚3969l=x²\’Ö &loss/total_loss‚42.481252198866855 0loss/FRt3/assignment_loss‚3.6790145436922708 2loss/SRqqt3/assignment_loss‚1.8221271609937824 2loss/SRqqt4/assignment_loss‚1.7455145247423878 0loss/FBt1/assignment_loss‚0.1552770928431412 /loss/FRt2/detection_loss‚2.2013450605528697 1loss/SRqqt1/detection_loss‚1.0830420812680608 2loss/SRqqt1/assignment_loss‚1.1181799290731314 0loss/FBt2/assignment_loss‚0.2873691103554198 1loss/SRqqt4/detection_loss‚1.8280315657074635 _runtime‚3969.252362233 /loss/FRt4/assignment_loss‚2.763711371592113 /loss/FRt1/detection_loss‚1.9903323887644901 /loss/FBt1/detection_loss‚0.6132885756656185 /loss/FBt2/detection_loss‚2.0427270999976566 epoch‚3 trainer/global_step‚4749 _step‚388 /loss/FRt1/assignment_loss‚1.305403540963712 1loss/SRqqt3/detection_loss‚2.3867671860230937 .loss/FBt3/detection_loss‚2.883189539114634 $ _timestamp‚1.788441529207743e+09 .loss/FRt3/detection_loss‚2.578808248043452 /loss/FBt4/detection_loss‚2.8909798935055733 2loss/SRqqt2/assignment_loss‚1.5271482587171097 1loss/SRqqt2/detection_loss‚1.6932573596636453 0loss/FRt2/assignment_loss‚2.8732103766227257 /loss/FRt4/detection_loss‚2.4967140704393387 0loss/FBt3/assignment_loss‚0.2664883444667794 1loss/FBt4/assignment_loss‚0.24932487605838105å””a“Û /loss/FRt4/assignment_loss‚2.763711371592113 /loss/FRt1/detection_loss‚1.9903323887644901 /loss/FBt1/detection_loss‚0.6132885756656185 /loss/FBt2/detection_loss‚2.0427270999976566 epoch‚3 trainer/global_step‚4749 _step‚388 /loss/FRt1/assignment_loss‚1.305403540963712 1loss/SRqqt3/detection_loss‚2.3867671860230937 .loss/FBt3/detection_loss‚2.883189539114634 $ _timestamp‚1.788441529207743e+09 .loss/FRt3/detection_loss‚2.578808248043452 /loss/FBt4/detection_loss‚2.8909798935055733 2loss/SRqqt2/assignment_loss‚1.5271482587171097 1loss/SRqqt2/detection_loss‚1.6932573596636453 0loss/FRt2/assignment_loss‚2.8732103766227257 /loss/FRt4/detection_loss‚2.4967140704393387 0loss/FBt3/assignment_loss‚0.2664883444667794 1loss/FBt4/assignment_loss‚0.24932487605838105 &loss/total_loss‚42.481252198866855 0loss/FRt3/assignment_loss‚3.6790145436922708 2loss/SRqqt3/assignment_loss‚1.8221271609937824 2loss/SRqqt4/assignment_loss‚1.7455145247423878 0loss/FBt1/assignment_loss‚0.1552770928431412 /loss/FRt2/detection_loss‚2.2013450605528697 1loss/SRqqt1/detection_loss‚1.0830420812680608 2loss/SRqqt1/assignment_loss‚1.1181799290731314 0loss/FBt2/assignment_loss‚0.2873691103554198 1loss/SRqqt4/detection_loss‚1.8280315657074635 _runtime‚3969.252362233„œ:†Ÿ”‚(:” ¾çåÔ͉µÉ gpu.0.gpu‚55.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.294' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.258799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ÿÛ=•‚(:² ¾çåÔ–ùºË network.recv‚ 1.49347e+06% disk./.usageGB‚1108.6084899902344 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875 network.sent‚ 1.0941149e+07 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚35141.2412109375 disk.dm-1.out‚ 154.5703125# proc.memory.rssMB‚ 2630.44140625" disk.dm-0.out‚35141.2412109375 disk.rbd2.out‚ 263.69140625 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 proc.cpu.threads‚17) disk./.usagePercent‚74.39964594772432 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 154.5703125 cpu‚25.269912288783782 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚35141.2412109375 disk.sda3.out‚ 154.5703125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.04296875) proc.memory.availableMB‚ 2034.62109375$ memory_percent‚87.58165836334229* proc.memory.percent‚16.054940223693848 disk.sda1.in‚0'²Ò~”–‚(:‰ ÍçåÔ¸²ŒÉ gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚131.464' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Àb.A—‚(:¶ ÍçåÔ†ßöÊ disk.drbd1777.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-1.out‚ 155.0390625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd1777.in‚0 cpu‚25.269098089396078 disk.sda3.out‚ 155.0390625 disk.rbd0.in‚0* proc.memory.percent‚16.054940223693848) disk./.usagePercent‚74.39965183589761% disk./.usageGB‚1108.6085777282715 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 2034.23828125 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚266.05908203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.sda.out‚ 155.0390625 disk.sda1.in‚0$ memory_percent‚87.58399486541748# proc.memory.rssMB‚ 2630.44140625 network.sent‚ 1.1049972e+07 network.recv‚ 1.500104e+06 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1754.04296875% disk.nvme0n1.out‚35141.6044921875' disk.nvme0c0n1.out‚35141.6044921875" disk.dm-0.out‚35141.6044921875I`rkž˜‚(:“ ÜçåÔž€áÈ gpu.0.gpu‚82.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.993' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.19859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÉgðA™‚(:¶ ÜçåÔ€¼¬Ë# disk.dm-0.out‚35239.05810546875 disk.dm-1.out‚ 155.5234375 disk.dm-0.in‚ 1754.04296875* proc.memory.percent‚16.054940223693848) disk./.usagePercent‚74.39965823608597% disk./.usageGB‚1108.6086730957031 disk.rbd1.out‚ 1.83984375 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚35239.05810546875 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2630.44140625 disk.rbd2.in‚0$ memory_percent‚87.58254051208496 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 network.recv‚ 1.50429e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 155.5234375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 network.sent‚ 1.1055779e+07 disk.sda2.out‚0! disk.nvme0n1.in‚ 1754.04296875( proc.memory.availableMB‚ 2034.4765625 disk.sda.out‚ 155.5234375 disk.rbd3.out‚0! disk.rbd2.out‚268.20751953125 disk.sda.in‚0.5625( disk.nvme0c0n1.out‚35239.05810546875 cpu‚25.26833969866391‚¥¯2š- _wandbruntime‚4008  _runtime‚4008dÀ©ƒÇ›Á & lr-AdamW/pg1‚0.0002924225571025475 & lr-AdamW/pg2‚0.0002924225571025475 trainer/global_step‚4799 % _timestamp‚1.7884415681238146e+09 _runtime‚4008.168508594 _step‚389M¢OíÌœÆ _step‚389 & lr-AdamW/pg1‚0.0002924225571025475 & lr-AdamW/pg2‚0.0002924225571025475 trainer/global_step‚4799 % _timestamp‚1.7884415681238146e+09 _runtime‚4008.168508594…ydÅe2- _wandbruntime‚4008  _runtime‚4008}KžúO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70399317 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8884705248768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45641934 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78811298 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12185176742912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884415681245968e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523701528914944 _runtime‚4008.169475306 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89208708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89228742 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8872603996950 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚532086530448384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8887305458176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚467417413349376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚540971209305600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70399323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚467417492074496 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _step‚390 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚532086504056832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532571509199258 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70399317 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89208708 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚540973835906560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚540973835906560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12185126152192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚532086530448384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33169361 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8870005712282 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70399323 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8884705248768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚532086504056832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8887305458176 trainer/global_step‚4799 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚540971209305600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89228742 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78810832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248%Ò&(ŸÿO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8887305458176 trainer/global_step‚4799 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚540971209305600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89228742 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78810832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70399317 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8884705248768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45641934 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78811298 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12185176742912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884415681245968e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523701528914944 _runtime‚4008.169475306 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89208708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89228742 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8872603996950 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚532086530448384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8887305458176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚467417413349376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚540971209305600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70399323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚467417492074496 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _step‚390 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚532086504056832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532571509199258 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70399317 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89208708 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚540973835906560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚540973835906560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12185126152192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚532086530448384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33169361 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8870005712282 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70399323 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8884705248768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚532086504056832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574†=ò8i2 - _wandbruntime‚4008  _runtime‚4008t¹±þ '¡„N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89226942 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889137800192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45651133 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚467512366003200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78826544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70413559 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455324675250688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523807870089216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8886537577472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33175861 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚391 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8871835054950 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8886537577472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70413553 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12187662618112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8874433339826 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455324703385088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70413553 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89247002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159640495 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78826994 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45651130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚541083607953920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720lÌ¡º¢‰N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89226942 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚532194470153728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89247002 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12187614137856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚532679679716198 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89247002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159640495 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78826994 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45651130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚541083607953920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89226942 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889137800192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45651133 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚467512366003200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78826544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_Ïj®¬Ubatch_end/allocation.large_pool.allocated‚70413559 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455324675250688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523807870089216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8886537577472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33175861 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚391 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8871835054950 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8886537577472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70413553 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12187662618112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8874433339826 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455324703385088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70413553 E¯‚(:³ §èåÔúË™Ì disk.sda1.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.39967282851542 disk.sda1.in‚0) proc.memory.availableMB‚ 2018.37109375 proc.cpu.threads‚17 network.sent‚ 1.118736e+07 disk.drbd1777.in‚0 cpu‚25.31291816269527 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 158.18359375 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875% disk./.usageGB‚1108.6088905334473 disk.sda3.out‚ 158.18359375# disk.nvme0c0n1.in‚ 1754.04296875 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚0.5625# proc.memory.rssMB‚ 2646.08984375 disk.rbd2.in‚0 disk.rbd2.out‚281.4306640625 disk.sda2.out‚0$ disk.nvme0n1.out‚35338.720703125 network.recv‚ 1.527194e+06! disk.dm-0.out‚35338.720703125 disk.rbd1.in‚0 disk.sda.out‚ 158.18359375$ memory_percent‚87.68084049224854 disk.sda3.in‚0.5625& disk.nvme0c0n1.out‚35338.720703125* proc.memory.percent‚16.150450706481934>­¬2°- _wandbruntime‚4088  _runtime‚4088¯ di*±ãT QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚71459856 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2648 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚548670855305216 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8961525831142 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚539694470353408 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚539694470353408 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚71459861 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012864 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚540029680935998 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8978359964672 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23670554624 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚548670855305216 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚461687808678400 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚33598640 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚473989549456384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚531066182971392 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚115 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚539694446059008 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚46370759 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚33598753 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3300917248 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚79969512 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23670554624 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚531066206302208 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23670554624 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚548670855305216 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚461687808678400 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚33598640 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚473989549456384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚531066182971392 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚115 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚539694446059008 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚46370759 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚33598753 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3300917248 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚79969512 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23670554624 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚531066206302208 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0‰šÑUD2³- _wandbruntime‚4089  _runtime‚4089\}rq)´ëR GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8961998517030 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚2648 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚548747063109120 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8976883449344 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3300917248 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1574 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚539768228941312 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚531136184072192 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚71470750 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 128430592 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 $ _timestamp‚1.788441649434803e+09 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 _runtime‚ 4089.47990385 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚90265176 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8978834167808 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚90265176 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚33602429 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚461749791032832 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161735931 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚71470755 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚474051877583360 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1574 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚548745088096256 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚539768228941312 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12302104199168 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚71470755 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚71470750 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8963947319678 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161719952 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚531136207403008 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46378689 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚474052006013952 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23670554624 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚79981118 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚539768204646912 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133464 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2648 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8978834167808 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚90249202 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚113 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚540098182589222 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚115 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚46378687 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚33602316 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 110781952 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948802648 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚548747063109120 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 _step‚394 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚540100154722686 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8976883449344 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚79981003 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2648 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1574 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚12302214981120 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚548745088096256 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚461749773384192 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23670554624 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚539768204646912 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400.­Ákv)µðR QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161735931 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚71470755 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚474051877583360 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1574 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚548745088096256 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚539768228941312 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12302104199168 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚71470755 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚71470750 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8963947319678 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161719952 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚531136207403008 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46378689 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚474052006013952 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23670554624 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚79981118 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚539768204646912 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133464 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2648 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8978834167808 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚90249202 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚113 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚540098182589222 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚115 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚46378687 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚33602316 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 110781952 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948802648 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚548747063109120 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 _step‚394 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚540100154722686 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8976883449344 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚79981003 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2648 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1574 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚12302214981120 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚548745088096256 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚461749773384192 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23670554624 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚539768204646912 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8961998517030 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚2648 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚548747063109120 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8976883449344 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3300917248 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1574 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚539768228941312 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚531136184072192 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚71470750 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 128430592 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 $ _timestamp‚1.788441649434803e+09 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 _runtime‚ 4089.47990385 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚90265176 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8978834167808 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚90265176 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚33602429 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚461749791032832 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464Š°Òí¶‚(:’ ¶èåÔ°ï—É gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚42.0 gpu.0.powerWatts‚197.373' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.474599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ÖÇ4·‚(:© ¶èåÔÄø§Ì disk.rbd0.out‚0 disk.drbd1777.in‚0$ memory_percent‚88.11769485473633% disk./.usageGB‚1108.6097793579102& disk.nvme0c0n1.out‚35436.310546875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 158.46875 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2718.3359375* proc.memory.percent‚16.591405868530273 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 158.46875 disk.sda2.out‚0 network.sent‚ 1.1192517e+07 disk.rbd2.out‚283.5166015625 disk.sda2.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.39973247827089 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.in‚0.5625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625$ disk.nvme0n1.out‚35436.310546875! disk.dm-0.out‚35436.310546875 network.recv‚ 1.530927e+06 disk.sda.in‚0.5625' proc.memory.availableMB‚ 1946.796875 cpu‚25.328639143672707 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 158.46875‰Jñ2¸- _wandbruntime‚4102  _runtime‚4102‚•`T¿¹Û¦ =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5492430925369263 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6848544478416443 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008547062985599041 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8543679118156433 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19887909293174744 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 epoch‚3 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45458483695983887 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5290387272834778 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4978910982608795 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5360252261161804 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 jet/accuracy_7_of_7‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5568452477455139 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20138433575630188 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_4_of_5‚1 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.512368381023407 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.038860902190208435 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.038888849318027496 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8163300156593323 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.472988486289978 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1505693644285202 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5428931713104248 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5833333134651184 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6555242538452148 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4155057668685913 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884416628248274e+09 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.517780601978302 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48846635222435 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24612495303153992 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43023255467414856 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5511178970336914 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13695767521858215 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1786234825849533 particle/accuracy_1_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5272059440612793 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_6_of_6‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5948218703269958 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1953912079334259 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.49122804403305054 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1787271350622177 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5072564482688904 ?&Purity/3FŒ—¸£xRt*SRqqt0FBt/event_proportion‚0.007254464086145163 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5487999320030212 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5301458239555359 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31790122389793396 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26317843794822693 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896097183227539 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5373837947845459 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_6‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992238879203796 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12729914486408234 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4462718963623047 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7626749873161316 particle/accuracy_6_of_6‚1 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.475926011800766 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18155339360237122 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911189675331116 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24313199520111084 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10597041249275208 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5382259488105774 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1427532136440277 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900886416435242 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5559523701667786 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4300000071525574 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1770833432674408 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 jet/accuracy_4_of_7‚0 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866974949836731 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08189655095338821 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9891601204872131 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999403953552 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555555045604706 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15939553081989288 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23923256993293762 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9876792430877686 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11566171050071716 jet/accuracy_5_of_5‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 _step‚395 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899967312812805 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.032911356538534164 *jet/accuracy_1_of_3‚0.8719262480735779 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5382259488105774 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 _runtime‚4102.871428003 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08696254342794418 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37239861488342285 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.033192526549100876 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.506944477558136 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5584874749183655 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10153256356716156 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07020662724971771 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.48840147256851196 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3362351059913635 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4785403311252594 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14874063432216644 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_2‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13812421262264252 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1288718730211258 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5682953000068665 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1750706285238266 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4922412931919098 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 trainer/global_step‚4835 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27415594458580017 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4463161826133728 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11754385381937027 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9576237797737122 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5867362022399902 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.2964675724506378 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993650913238525 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9574180841445923 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905905723571777 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.03846153989434242 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993650913238525 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1392587423324585 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02301792986690998 +jet/accuracy_2_of_4‚0.39874520897865295 +jet/accuracy_3_of_4‚0.07482451945543289 particle/accuracy_1_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5833333730697632 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00833333283662796 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13723060488700867 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937641620635986 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5370948910713196 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971655011177063 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3280901610851288 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28388747572898865 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17676766216754913 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19187670946121216 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2935517132282257 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12717637419700623 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4353555738925934 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09313888102769852 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996001124382019 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13742199540138245 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3239032030105591 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3859863877296448 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911456108093262 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2888889014720917 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121211737394333 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5597036480903625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28091350197792053 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5231481790542603 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9869408011436462 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5155414342880249 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14609023928642273 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5552113652229309 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4606941342353821 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1505693644285202 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4551282227039337 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12933816015720367 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1178496703505516 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3749999701976776 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985221803188324 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 jet/accuracy_2_of_7‚0 particle/accuracy_6_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13500972092151642 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48534515500068665 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5684027075767517 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3043155074119568 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03779762238264084 )jet/accuracy_1_of_4‚0.926228404045105 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9911496043205261 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5583333373069763 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23769040405750275 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5318676829338074 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882699251174927 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.13596491515636444 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5167161822319031 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3319014012813568 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0795564204454422 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5406863689422607 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9960795044898987 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024206530302762985 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5291199088096619 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33098748326301575 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9727011322975159 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24624116718769073 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702128499746323 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015910809859633446 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5211508870124817 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20987656712532043 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12248243391513824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5235249400138855 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2663980722427368 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5469473600387573 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_2_of_3‚0.24128475785255432 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.50471031665802 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1878681778907776 particle/accuracy_1_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2114146202802658 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13355478644371033 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4829557240009308 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25439706444740295 particle/accuracy_3_of_4‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5052188634872437 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.14089222252368927 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 'particle/f_score‚0.3884493112564087 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 jet/accuracy_1_of_7‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4739105999469757 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08814149349927902 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11062926054000854 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5243570804595947 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023149294778704643 particle/accuracy_2_of_3‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12183909118175507 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *loss/val_total_loss‚43.154593590455654 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006698320619761944 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890400171279907 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5649309754371643 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4878811240196228 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970888495445251 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4975278079509735 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3780101537704468 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9778911471366882 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9921388626098633 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10493826866149902 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5137996077537537 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.44708433747291565 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5511337518692017 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10907817631959915 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9774001240730286 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4446107745170593 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9955077171325684 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23936781287193298 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4903903603553772 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.018881209194660187 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4817870557308197 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46931448578834534 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17436327040195465 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_6‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01967177540063858 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17680180072784424 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01899743638932705 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5505699515342712 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30659720301628113 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2250000387430191 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.545711100101471 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17790016531944275 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0012210012646391988 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0[žÂ­Oºà¦ 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5833333134651184 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6555242538452148 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4155057668685913 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884416628248274e+09 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.517780601978302 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48846“Iù635222435 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24612495303153992 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43023255467414856 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5511178970336914 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13695767521858215 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1786234825849533 particle/accuracy_1_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5272059440612793 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_6_of_6‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5948218703269958 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1953912079334259 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.49122804403305054 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1787271350622177 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5072564482688904 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5487999320030212 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5301458239555359 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31790122389793396 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26317843794822693 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896097183227539 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5373837947845459 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_6‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992238879203796 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12729914486408234 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4462718963623047 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7626749873161316 particle/accuracy_6_of_6‚1 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.475926011800766 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18155339360237122 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9911189675331116 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24313199520111084 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10597041249275208 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5382259488105774 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1427532136440277 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900886416435242 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5559523701667786 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4300000071525574 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1770833432674408 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 jet/accuracy_4_of_7‚0 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866974949836731 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08189655095338821 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9891601204872131 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999403953552 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555555045604706 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15939553081989288 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23923256993293762 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9876792430877686 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11566171050071716 jet/accuracy_5_of_5‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 _step‚395 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899967312812805 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.032911356538534164 *jet/accuracy_1_of_3‚0.8719262480735779 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5382259488105774 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 _runtime‚4102.871428003 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08696254342794418 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37239861488342285 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.033192526549100876 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.506944477558136 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5584874749183655 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10153256356716156 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07020662724971771 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.48840147256851196 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3362351059913635 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4785403311252594 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14874063432216644 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_2‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13812421262264252 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1288718730211258 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5682953000068665 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1750706285238266 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4922412931919098 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 trainer/global_step‚4835 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27415594458580017 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4463161826133728 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11754385381937027 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9576237797737122 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5867362022399902 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.2964675724506378 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993650913238525 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9574180841445923 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905905723571777 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.03846153989434242 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993650913238525 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1392587423324585 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02301792986690998 +jet/accuracy_2_of_4‚0.39874520897865295 +jet/accuracy_3_of_4‚0.07482451945543289 particle/accuracy_1_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5833333730697632 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00833333283662796 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13723060488700867 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937641620635986 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5370948910713196 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971655011177063 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3280901610851288 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28388747572898865 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17676766216754913 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19187670946121216 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2935517132282257 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12717637419700623 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4353555738925934 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09313888102769852 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996001124382019 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13742199540138245 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3239032030105591 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3859863877296448 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911456108093262 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2888889014720917 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121211737394333 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5597036480903625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28091350197792053 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5231481790542603 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9869408011436462 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5155414342880249 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14609023928642273 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5552113652229309 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4606941342353821 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1505693644285202 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4551282227039337 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12933816015720367 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1178496703505516 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3749999701976776 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985221803188324 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 jet/accuracy_2_of_7‚0 particle/accuracy_6_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13500972092151642 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48534515500068665 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5684027075767517 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3043155074119568 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03779762238264084 )jet/accuracy_1_of_4‚0.926228404045105 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9911496043205261 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5583333373069763 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23769040405750275 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5318676829338074 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882699251174927 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.13596491515636444 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5167161822319031 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3319014012813568 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0795564204454422 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5406863689422607 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9960795044898987 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024206530302762985 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5291199088096619 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33098748326301575 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9727011322975159 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24624116718769073 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702128499746323 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015910809859633446 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5211508870124817 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20987656712532043 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12248243391513824 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5235249400138855 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2663980722427368 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5469473600387573 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_2_of_3‚0.24128475785255432 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.50471031665802 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1878681778907776 particle/accuracy_1_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2114146202802658 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13355478644371033 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4829557240009308 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3819444477558136 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25439706444740295 particle/accuracy_3_of_4‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5052188634872437 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.14089222252368927 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 'particle/f_score‚0.3884493112564087 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 jet/accuracy_1_of_7‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4739105999469757 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08814149349927902 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11062926054000854 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5243570804595947 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023149294778704643 particle/accuracy_2_of_3‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12183909118175507 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *loss/val_total_loss‚43.154593590455654 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006698320619761944 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890400171279907 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5649309754371643 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4878811240196228 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970888495445251 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4975278079509735 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3780101537704468 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9778911471366882 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9921388626098633 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10493826866149902 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5137996077537537 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.44708433747291565 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5511337518692017 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10907817631959915 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9774001240730286 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4446107745170593 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9955077171325684 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23936781287193298 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4903903603553772 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.018881209194660187 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4817870557308197 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46931448578834534 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17436327040195465 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_6‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01967177540063858 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17680180072784424 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01899743638932705 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5505699515342712 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30659720301628113 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2250000387430191 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.545711100101471 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17790016531944275 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0012210012646391988 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5492430925369263 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6848544478416443 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008547062985599041 ,&Purity/2FRt2SRqqt-¡ú 3FBt/event_proportion‚0 (particle/accuracy‚0.8543679118156433 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19887909293174744 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 epoch‚3 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45458483695983887 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5290387272834778 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4978910982608795 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5360252261161804 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 jet/accuracy_7_of_7‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5568452477455139 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20138433575630188 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_4_of_5‚1 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.512368381023407 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.038860902190208435 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.038888849318027496 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8163300156593323 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.472988486289978 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1505693644285202 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5428931713104248 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0‹ŸÀ‘»‚(:† ÅèåÔúÌüÈ gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚32.0 gpu.0.powerWatts‚77.001' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.4002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00­H]0¼‚(:¥ ÅèåÔþŽ¸Ë% disk./.usageGB‚1108.6097831726074 disk.sda3.in‚0.5625 disk.dm-0.out‚ 36808.78125 disk.drbd1777.out‚0 disk.sda2.in‚0 network.recv‚ 1.537549e+06 disk.dm-1.out‚ 159.0234375( proc.memory.availableMB‚ 2889.9765625# proc.memory.rssMB‚ 3581.58203125 disk.sda3.out‚ 159.0234375 disk.rbd1.out‚ 1.98828125 disk.sda.in‚0.5625 disk.sda1.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.out‚ 36808.78125 disk.rbd3.out‚0 network.sent‚ 1.1300906e+07) disk./.usagePercent‚74.39973273427843 disk.dm-1.in‚0.5625 disk.rbd2.out‚287.8994140625 disk.sda2.out‚0$ memory_percent‚82.36098289489746 disk.dm-0.in‚ 1754.04296875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 cpu‚25.336462828563892 disk.nvme0n1.out‚ 36808.78125 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.sda.out‚ 159.0234375* proc.memory.percent‚21.860241889953613:õsê2½- _wandbruntime‚4123  _runtime‚4123ó2æȾ ' lr-AdamW/pg1‚0.00029000507525514014 ' lr-AdamW/pg2‚0.00029000507525514014 trainer/global_step‚4849 % _timestamp‚1.7884416836643212e+09 _runtime‚ 4123.70898134 _step‚396^ç”Í¿Ç ' lr-AdamW/pg1‚0.00029000507525514014 ' lr-AdamW/pg2‚0.00029000507525514014 trainer/global_step‚4849 % _timestamp‚1.7884416836643212e+09 _runtime‚ 4123.70898134 _step‚396ŒIâé2À- _wandbruntime‚4123  _runtime‚4123i7çïÿ'ÁùO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542419914306188 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚542130276259840 _step‚397 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71797439 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90580179 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚542130302651392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476098232251392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12341572348928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71797439 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8993881039880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚551139130746880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542422538018824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚551136504145920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9006227886080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9008828095488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71797445 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚405 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33737878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476098166109184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71797445 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9008828095488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 trainer/global_step‚4849 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9006227886080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788441683665034e+09 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90600213 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80339195 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463756621894656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38007808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46601317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33738280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90580179 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚551139130746880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533428631550976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚542130302651392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚402 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12341610356736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90580179 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚542130302651392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476098232251392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12341572348928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71797439 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8993881039880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚551139130746880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542422538018824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚551136504145920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9006227886080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9008828095488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71797445 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚405 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33737878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476098166109184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71797445 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9008828095488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 trainer/global_step‚4849 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9006227886080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788441683665034e+09 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90600213 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80339195 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463756621894656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38007808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46601317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33738280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90580179 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚551139130746880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533428631550976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚542130302651392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚402 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12341610356736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533534972725248 _runtime‚4124.443779312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162410088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12344135312896 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90598413 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551248902794240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚533534998153216 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542238242356736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9010660437504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚476193117126656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚4849 _step‚398 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551246276179968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8995710382756 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚379 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80355090 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71811681 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71811681 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551248902794240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46610516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚542238215965184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 262661427—\ƒ)e 2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 % _timestamp‚1.7884416843983014e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90618473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33744577 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542530708535972 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8995710382756 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚379 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80355090 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71811681 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71811681 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551248902794240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46610516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚542238215965184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 % _timestamp‚1.7884416843983014e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90618473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33744577 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542530708535972 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚533534972725248 _runtime‚4124.443779312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162410088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12344135312896 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90598413 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551248902794240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚533534998153216 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542238242356736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400Ž‘Åð2Æ- _wandbruntime‚4124  _runtime‚4124È8H`VÇÐ trainer/global_step‚4849 _runtime‚ 4124.44494046 /loss/FRt3/assignment_loss‚3.447560374438763 2loss/SRqqt1/assignment_loss‚0.9946726142626369 1loss/FBt3/assignment_loss‚0.08408683761954308 /loss/FBt1/detection_loss‚0.8331950393694569 $loss/total_loss‚39.9289022985959 /loss/FRt2/assignment_loss‚2.804735412200292 2loss/SRqqt3/assignment_loss‚1.6834949252661318 2loss/SRqqt4/assignment_loss‚1.2958611507041782 1loss/FBt1/assignment_loss‚0.11922212207624246 1loss/SRqqt1/detection_loss‚0.9841609715173641 /loss/SRqqt4/detection_loss‚2.03031070209017 0loss/SRqqt3/detection_loss‚2.331998659297824 epoch‚4 1loss/FBt4/assignment_loss‚0.09381856356048957 0loss/FRt4/assignment_loss‚2.4439649216153403 1loss/SRqqt2/assignment_loss‚1.024668374331668 0loss/FBt2/assignment_loss‚0.1013928091069829 1loss/SRqqt2/detection_loss‚1.6024712235666811 /loss/FBt3/detection_loss‚3.1833611011505125 _step‚399 /loss/FRt2/detection_loss‚2.2946705600867667 .loss/FRt3/detection_loss‚2.323374736310236 .loss/FBt2/detection_loss‚2.171948936250475 $ _timestamp‚1.788441684400366e+09 /loss/FRt1/detection_loss‚2.1277681424149444 0loss/FRt1/assignment_loss‚1.3251800622258867 /loss/FRt4/detection_loss‚2.5908823155544023 .loss/FBt4/detection_loss‚2.036101743578911»†ð~[ÈÕ .loss/FBt2/detection_loss‚2.171948936250475 $ _timestamp‚1.788441684400366e+09 /loss/FRt1/detection_loss‚2.1277681424149444 0loss/FRt1/assignment_loss‚1.3251800622258867 /loss/FRt4/detection_loss‚2.5908823155544023 .loss/FBt4/detection_loss‚2.036101743578911 trainer/global_step‚4849 _runtime‚ 4124.44494046 /loss/FRt3/assignment_loss‚3.447560374438763 2loss/SRqqt1/assignment_loss‚0.9946726142626369 1loss/FBt3/assignment_loss‚0.08408683761954308 /loss/FBt1/detection_loss‚0.8331950393694569 $loss/total_loss‚39.9289022985959 /loss/FRt2/assignment_loss‚2.804735412200292 2loss/SRqqt3/assignment_loss‚1.6834949252661318 2loss/SRqqt4/assignment_loss‚1.2958611507041782 1loss/FBt1/assignment_loss‚0.11922212207624246 1loss/SRqqt1/detection_loss‚0.9841609715173641 /loss/SRqqt4/detection_loss‚2.03031070209017 0loss/SRqqt3/detection_loss‚2.331998659297824 epoch‚4 1loss/FBt4/assignment_loss‚0.09381856356048957 0loss/FRt4/assignment_loss‚2.4439649216153403 1loss/SRqqt2/assignment_loss‚1.024668374331668 0loss/FBt2/assignment_loss‚0.1013928091069829 1loss/SRqqt2/detection_loss‚1.6024712235666811 /loss/FBt3/detection_loss‚3.1833611011505125 _step‚399 /loss/FRt2/detection_loss‚2.2946705600867667 .loss/FRt3/detection_loss‚2.323374736310236Æ&é”É‚(:‰ ÔèåÔé´‹É gpu.0.gpu‚80.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚44.0 gpu.0.powerWatts‚230.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.1152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ÐK5Ê‚(:ª ÔèåÔ“ÚõÊ network.recv‚ 1.544216e+06 disk.drbd1777.out‚0 disk.sda3.out‚ 159.390625 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 159.390625 disk.drbd1777.in‚0 disk.rbd2.out‚292.0712890625! disk.nvme0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 3747.70703125* proc.memory.percent‚16.636300086975098 disk.rbd3.out‚0 cpu‚25.335013027173023 disk.rbd2.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1754.04296875% disk.nvme0c0n1.out‚41163.77734375# disk./.usageGB‚1108.60986328125 disk.sda.out‚ 159.390625 disk.rbd3.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2725.69140625 proc.cpu.threads‚17) disk./.usagePercent‚74.39973811043666 disk.rbd1.in‚0 disk.rbd1.out‚ 1.98828125# disk.nvme0n1.out‚41163.77734375$ memory_percent‚77.12581157684326 disk.dm-0.out‚41163.77734375 disk.sda2.out‚0 disk.sda3.in‚0.5625 network.sent‚ 1.1425076e+07]eÝ“Ë‚(:ˆ ãèåÔ«ÅëÈ gpu.0.gpu‚73.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚43.0 gpu.0.powerWatts‚95.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=—O¸DÌ‚(:¹ ãèåÔ–Þ÷Ë disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.75390625 network.recv‚ 1.551061e+06 disk.sda.in‚0.5625 disk.sda3.in‚0.5625) proc.memory.percent‚16.63668155670166 network.sent‚ 1.1534001e+07 disk.dm-1.out‚ 160.1328125 disk.sda.out‚ 160.1328125) proc.memory.availableMB‚ 3747.97265625 cpu‚25.333974276689688 disk.sda3.out‚ 160.1328125# disk.nvme0c0n1.in‚ 1754.04296875# disk.dm-0.out‚41263.53759765625 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚41263.53759765625 disk.rbd3.in‚0 disk.drbd1777.out‚0$ memory_percent‚77.12419033050537 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.sda2.in‚0& disk.nvme0n1.out‚41263.53759765625 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.0205078125 disk.rbd2.in‚0) disk./.usagePercent‚74.39984870569144% disk./.usageGB‚1108.6115112304688 disk.rbd2.out‚295.4462890625»p0“Í‚(:ˆ òèåÔ‚¶ÿÈ gpu.0.gpu‚57.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚216.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼a]þD΂(:¹ òèåÔç¬ÏÌ disk.sda1.in‚0 disk.sda3.in‚0.5625) disk./.usagePercent‚74.39984870569144 disk.rbd1.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚41263.91845703125$ memory_percent‚77.12557315826416) proc.memory.percent‚16.63668155670166 cpu‚25.332887244286287 proc.cpu.threads‚17 disk.rbd2.out‚300.6923828125# disk.nvme0c0n1.in‚ 1754.04296875 network.sent‚ 1.1539454e+07 network.recv‚ 1.555199e+06 disk.sda.out‚ 160.4921875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 3747.74609375 disk.sda.in‚0.5625 disk.rbd0.in‚0% disk./.usageGB‚1108.6115112304688 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 160.4921875 disk.rbd1.out‚ 2.0205078125 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.04296875& disk.nvme0n1.out‚41263.91845703125 disk.dm-0.in‚ 1754.04296875# disk.dm-0.out‚41263.91845703125 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 160.4921875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.75390625 disk.drbd1777.in‚0uú$2Ï- _wandbruntime‚4163  _runtime‚4163} 7yÇÐÁ & lr-AdamW/pg1‚0.0002875675354629099 & lr-AdamW/pg2‚0.0002875675354629099 trainer/global_step‚4899 % _timestamp‚1.7884417231511452e+09 _runtime‚4163.195831922 _step‚400Èåµ™ÌÑÆ trainer/global_step‚4899 % _timestamp‚1.7884417231511452e+09 _runtime‚4163.195831922 _step‚400 & lr-AdamW/pg1‚0.0002875675354629099 & lr-AdamW/pg2‚0.0002875675354629099ì] Â2Ò- _wandbruntime‚4163  _runtime‚4163Þä«ÿ'ÓùO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556625120262144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚547527287918592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556627746863104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81139308 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538745690264576 _runtime‚4163.196884104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81138863 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34078188 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9082763479820 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12468963682816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91515265 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47061120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72509239 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72509239 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468376024569856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9085361764488 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9097858735104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91515265 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91495231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468375996435456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547831077457032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4899 $ _timestamp‚1.788441723151914e+09eÄd½ÅÔþO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72509239 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72509239 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468376024569856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9085361764488 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9097858735104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91515265 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91495231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468375996435456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes™¦Xu?!.all.current‚ 2626600960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547831077457032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4899 $ _timestamp‚1.788441723151914e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556625120262144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚547527287918592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556627746863104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81139308 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538745690264576 _runtime‚4163.196884104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81138863 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34078188 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9082763479820 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12468963682816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _step‚401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91515265 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47061120‘œcØ2Õ- _wandbruntime‚4163  _runtime‚4163M4 'Ö„N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9102291286528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9102291286528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91533525 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚468468384488960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91513465 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480939856199168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚402 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12471513899008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72523475 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81154791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547936624261336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47070316 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547939247974180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91513465 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚4899 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81155218 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚547635227623936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47070313 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468468412623360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884417238868122e+09 _runtime‚4163.932004116 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480939926522368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12471471710208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚538852031438848 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72523481 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚547635227623936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556734892296192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34084478 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574ˆSè'׉N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12471471710208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚538852031438848 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72523481 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚547635227623936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556734892296192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34084478 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164057006 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9102291286528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9102291286528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91533525 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚468468384488960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91513465 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480939856199168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚402 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12471513899008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72523475 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81154791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547936624261336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47070316 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547939247974180 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91513465 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚4899 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81155218 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚547635227623936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47070313 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468468412623360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884417238868122e+09 _runtime‚4163.932004116 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480939926522368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272’•h/¦2Ø- _wandbruntime‚4163  _runtime‚4163=Uˆ«YÙÓ 2loss/SRqqt3/assignment_loss‚1.4255925423010838 /loss/FRt2/detection_loss‚2.7409342527389526 /loss/FRt4/detection_loss‚2.2782116127865657 1loss/FBt4/assignment_loss‚0.26935160388662055 .loss/FBt2/detection_loss‚2.652314994070265 _step‚403 %loss/total_loss‚42.45460657053253 /loss/FRt3/assignment_loss‚3.007641396214885 0loss/FBt1/assignment_loss‚0.2223064518119725 1loss/SRqqt3/detection_loss‚2.2647650657698164 0loss/SRqqt4/detection_loss‚2.022907409071922 _runtime‚4163.933583962 $ _timestamp‚1.788441723888962e+09 0loss/FRt2/assignment_loss‚3.8416020902427466 1loss/FBt2/assignment_loss‚0.21961455788712353 /loss/FRt3/detection_loss‚2.7719868210054215 epoch‚4 0loss/FBt3/assignment_loss‚0.2638460421638394 0loss/SRqqt2/detection_loss‚2.128817825847202 /loss/FBt3/detection_loss‚1.9394192912361838 trainer/global_step‚4899 0loss/FRt1/assignment_loss‚1.2762799148376172 2loss/SRqqt1/assignment_loss‚1.2219203059584451 2loss/SRqqt4/assignment_loss‚1.5640927251428365 1loss/SRqqt1/detection_loss‚1.2636520631733488 .loss/FRt4/assignment_loss‚3.09402240387031 2loss/SRqqt2/assignment_loss‚1.5605212812622389 /loss/FRt1/detection_loss‚1.6588951770502787 /loss/FBt1/detection_loss‚0.5345692772498746 .loss/FBt4/detection_loss‚2.231341464952989J¯Pà^ÚØ 0loss/FBt3/assignment_loss‚0.2638460421638394 0loss/SRqqt2/detection_loss‚2.128817825847202 /loss/FBt3/detection_loss‚1.9394192912361838 trainer/global_step‚4899 0loss/FRt1/assignment_loss‚1.2762799148376172 2loss/SRqqt1/assignment_loss‚1.2219203059584451 2loss/SRqqt4/assignment_loss‚1.5640927251428365 1loss/SRqqt1/detection_loss‚1.2636520631733488 .loss/FRt4/assignment_loss‚3.09402240387031 2loss/SRqqt2/assignment_loss‚1.5605212812622389 /loss/FRt1/detection_loss‚1.6588951770502787 /loss/FBt1/detection_loss‚0.5345692772498746 .loss/FBt4/detection_loss‚2.231341464952989 2loss/SRqqt3/assignment_loss‚1.4255925423010838 /loss/FRt2/detection_loss‚2.7409342527389526 /loss/FRt4/detection_loss‚2.2782116127865657 1loss/FBt4/assignment_loss‚0.26935160388662055 .loss/FBt2/detection_loss‚2.652314994070265 _step‚403 %loss/total_loss‚42.45460657053253 /loss/FRt3/assignment_loss‚3.007641396214885 0loss/FBt1/assignment_loss‚0.2223064518119725 1loss/SRqqt3/detection_loss‚2.2647650657698164 0loss/SRqqt4/detection_loss‚2.022907409071922 _runtime‚4163.933583962 $ _timestamp‚1.788441723888962e+09 0loss/FRt2/assignment_loss‚3.8416020902427466 1loss/FBt2/assignment_loss‚0.21961455788712353 /loss/FRt3/detection_loss‚2.7719868210054215 epoch‚4“y7XS“Û‚(:ˆ éåÔÁ¡”É gpu.0.gpu‚77.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚260.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó<»Ü‚(:¯ éåÔü‚­Ë network.sent‚ 1.1545374e+07 disk.rbd3.out‚0# disk.nvme0n1.out‚41264.91796875 disk.sda3.in‚0.5625$ memory_percent‚77.12697982788086 proc.cpu.threads‚17% disk./.usageGB‚1108.6115684509277! disk.rbd2.out‚305.22021484375 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚41264.91796875) proc.memory.percent‚16.63668155670166 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.sda3.out‚ 160.9296875 disk.dm-1.out‚ 160.9296875) disk./.usagePercent‚74.39985254580446 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875' proc.memory.availableMB‚ 3747.515625 disk.sda.out‚ 160.9296875 disk.sda2.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚41264.91796875# proc.memory.rssMB‚ 2725.75390625 cpu‚25.33183249735733 network.recv‚ 1.5#>N359187e+06 disk.rbd1.out‚ 2.08349609375 disk.sda.in‚0.5625 disk.drbd1777.in‚0# disk.nvme0c0n1.in‚ 1754.04296875”¡"ï”Ý‚(:‰ éåÔµ±ÕÈ gpu.0.gpu‚71.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚234.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lX_ÚEÞ‚(:º éåÔä¥üË( proc.memory.availableMB‚ 3747.4296875) disk./.usagePercent‚74.39984870569144! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 161.3671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0) proc.memory.percent‚16.63668155670166 disk.sda3.in‚0.5625 network.recv‚ 1.565523e+06 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚41362.43798828125# disk.dm-0.out‚41362.43798828125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 2.08349609375 disk.sda.out‚ 161.3671875# disk.nvme0c0n1.in‚ 1754.04296875( disk.nvme0c0n1.out‚41362.43798828125! disk.rbd2.out‚310.52880859375 disk.sda.in‚0.5625 network.sent‚ 1.1653956e+07 disk.sda1.in‚0 disk.drbd1777.out‚0$ memory_percent‚77.12750434875488# proc.memory.rssMB‚ 2725.75390625 disk.rbd2.in‚0 cpu‚25.330821153583248 proc.cpu.threads‚17% disk./.usageGB‚1108.6115112304688 disk.sda2.out‚0 disk.dm-1.out‚ 161.3671875$òˆË“ß‚(:ˆ ŸéåÔ´É¨É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚42.0 gpu.0.powerWatts‚92.679' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ¿ÙÕ<à‚(:± ŸéåÔöܦË disk.sda1.out‚0% disk./.usageGB‚1108.6116523742676# disk.nvme0c0n1.in‚ 1754.04296875! disk.rbd2.out‚315.09521484375( proc.memory.availableMB‚ 3747.4453125 cpu‚25.32977361007625% disk.nvme0n1.out‚41362.9091796875 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.sda.out‚ 161.859375' disk.nvme0c0n1.out‚41362.9091796875" disk.dm-0.out‚41362.9091796875 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0$ memory_percent‚77.12740898132324) proc.memory.percent‚16.63668155670166 network.sent‚ 1.1660186e+07 network.recv‚ 1.569878e+06 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 2.146484375 disk.sda1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2725.75390625) disk./.usagePercent‚74.39985817797022! disk.nvme0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 161.859375 disk.sda3.out‚ 161.859375 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0×òs2á- _wandbruntime‚4202  _runtime‚4202À Ó]Åâ¿ _runtime‚4202.443218469 _step‚404 % lr-AdamW/pg1‚0.000285110445705496 % lr-AdamW/pg2‚0.000285110445705496 trainer/global_step‚4949 % _timestamp‚1.7884417623985374e+09 âz|ÊãÄ % lr-AdamW/pg1‚0.000285110445705496 % lr-AdamW/pg2‚0.000285110445705496 trainer/global_step‚4949 % _timestamp‚1.7884417623985374e+09 _runtime‚4202.443218469 _step‚404”Kk³¬2ä- _wandbruntime‚4202  _runtime‚42020¢ÙJ(åúO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9174244204428 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92430317 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92430317 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92410283 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 trainer/global_step‚4949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚552924246794240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485591661173760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12596304265216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73221039 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47520920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562113736378368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562113736378368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73221039 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9189489584128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9189489584128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472995427245056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472995399110656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562116362979328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚552924273185792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _runtime‚4202.444310176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚552924273185792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92410283 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553236993182604 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚552924246794240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚553239616895240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚544062774406144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47520917 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562116362979328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34416054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚09—ýX(æÿO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472995427245056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472995399110656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165631322 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562116362979328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚552924273185792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _runtime‚4202.444310176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚552924273185792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92410283 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553236993182604 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚552924246794240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚553239616895240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚544062774406144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47520917 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562116362979328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34416054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9174244204428 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92430317 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92430317 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92410283 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 trainer/global_step‚4949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚552924246794240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485591661173760 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12596304265216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73221039 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47520920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562113736378368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562113736378368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73221039 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9189489584128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9189489584128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚42202112•ïJô2ç- _wandbruntime‚4203  _runtime‚4203k*T!è„N ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚544169115580416 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485686662633472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚405 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81953465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4949 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81953060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12598805146112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92428517 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485686592310272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473087815298560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34423349 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚406 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9191321912832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562223508412416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚553032186499584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12598847334912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47530116 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73235275 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553347787412388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92428517 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562226135026688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9191321912832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9178671831972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9176073547096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚553032212891136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚553345163699544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segm료¶ent.large_pool.allocated‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 _runtime‚4203.181776695 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73235281 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473087787164160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34422947 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574[a¥y'é‰N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473087815298560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34423349 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚406 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9191321912832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562223508412416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚553032186499584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12598847334912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47530116 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73235275 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553347787412388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92428517 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562226135026688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9191321912832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9178671831972 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9176073547096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚553032212891136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚553345163699544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 _runtime‚4203.181776695 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73235281 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473087787164160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34422947 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚544169115580416 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485686662633472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚405 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81953465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4949 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81953060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12598805146112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92428517 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485686592310272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552–mãŠ2ê- _wandbruntime‚4203  _runtime‚4203Õßü¯`ëÚ /loss/FRt3/detection_loss‚2.3305368480229527 1loss/SRqqt2/detection_loss‚2.0976784932307706 % _timestamp‚1.7884417631388476e+09 _step‚407 /loss/FRt1/detection_loss‚1.5008509695116017 /loss/FRt2/detection_loss‚2.4153465174815874 .loss/FRt4/detection_loss‚2.450422747402775 epoch‚4 2loss/SRqqt1/assignment_loss‚0.8782034165018963 1loss/SRqqt4/detection_loss‚2.0316393867481586 0loss/FRt2/assignment_loss‚3.2062088698148727 0loss/FRt3/assignment_loss‚3.1560499639738175 0loss/FRt4/assignment_loss‚3.1702995215143477 2loss/SRqqt2/assignment_loss‚1.4349977716039388 .loss/FBt3/detection_loss‚2.948376327753067 1loss/SRqqt1/detection_loss‚1.1015962701704767 1loss/SRqqt3/detection_loss‚2.4546195147810757 /loss/FBt4/detection_loss‚3.7191845893859865 /loss/FRt1/assignment_loss‚1.139517926507526 1loss/SRqqt4/assignment_loss‚1.603711700066924 1loss/FBt1/assignment_loss‚0.12126812923456205 1loss/FBt4/assignment_loss‚0.21152574294246734 trainer/global_step‚4949 /loss/FBt1/detection_loss‚0.4490159059118014 &loss/total_loss‚42.480711396328104 _runtime‚4203.183502616 2loss/SRqqt3/assignment_loss‚1.2585379944963229 1loss/FBt2/assignment_loss‚0.06596221053041518 1loss/FBt3/assignment_loss‚0.01901927536043028 /loss/FBt2/detection_loss‚2.7161413033803306Øf|eìß 2loss/SRqqt1/assignment_loss‚0.8782034165018963 1loss/SRqqt4/detection_loss‚2.0316393867481586 0loss/FRt2/assignment_loss‚3.2062088698148727 0loss/FRt3/assignment_loss‚3.1560499639738175 0loss/FRt4/assignment_loss‚3.1702995215143477 2loss/SRqqt2/assignment_loss‚1.4349977716039388 .loss/FBt3/detection_loss‚2.948376327753067 1loss/SRqqt1/detection_loss‚1.1015962701704767 1loss/SRqqt3/detection_loss‚2.4546195147810757 /loss/FBt4/detection_loss‚3.7191845893859865 /loss/FRt1/assignment_loss‚1.139517926507526 1loss/SRqqt4/assignment_loss‚1.603711700066924 1loss/FBt1/assignment_loss‚0.12126812923456205 1loss/FBt4/assignment_loss‚0.21152574294246734 trainer/global_step‚4949 /loss/FBt1/detection_loss‚0.4490159059118014 &loss/total_loss‚42.480711396328104 _runtime‚4203.183502616 2loss/SRqqt3/assignment_loss‚1.2585379944963229 1loss/FBt2/assignment_loss‚0.06596221053041518 1loss/FBt3/assignment_loss‚0.01901927536043028 /loss/FBt2/detection_loss‚2.7161413033803306 /loss/FRt3/detection_loss‚2.3305368480229527 1loss/SRqqt2/detection_loss‚2.0976784932307706 % _timestamp‚1.7884417631388476e+09 _step‚407 /loss/FRt1/detection_loss‚1.5008509695116017 /loss/FRt2/detection_loss‚2.4153465174815874 .loss/FRt4/detection_loss‚2.450422747402775 epoch‚4—›òQ¸Ÿí‚(:” ®éåÔݺ¨É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚43.0 gpu.0.powerWatts‚147.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.492599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÿÃ7î‚(:¬ ®éåÔ§ÍÖË disk.dm-0.in‚ 1754.04296875( proc.memory.availableMB‚ 3746.2109375 disk.dm-1.in‚0.5625 network.sent‚ 1.1665444e+07% disk./.usageGB‚1108.6116523742676 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚74.39985817797022 disk.sda2.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.04296875% disk.nvme0c0n1.out‚41460.42578125 disk.rbd3.out‚0 network.recv‚ 1.573584e+06 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2725.75390625! disk.rbd2.out‚320.27880859375 disk.sda.out‚ 162.328125 disk.sda3.out‚ 162.328125$ memory_percent‚77.13494300842285) proc.memory.percent‚16.63668155670166 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚25.328838635652275 disk.rbd2.in‚0 disk.dm-0.out‚41460.42578125 disk.dm-1.out‚ 162.328125 proc.cpu.threads‚17 disk.sda2.in‚0# disk.nvme0n1.out‚41460.42578125 disk.rbd1.out‚ 2.146484375ølãqŸï‚(:” ½éåÔÉî•É gpu.0.gpu‚81.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚136.165' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.232999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`²v³<ð‚(:± ½éåÔšá×Ê disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 network.recv‚ 1.580157e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.04296875! disk.dm-0.out‚41460.896484375 disk.dm-1.in‚0.5625$ memory_percent‚77.13584899902344) disk./.usagePercent‚74.39986073804555 disk.sda1.in‚0$ disk.nvme0n1.out‚41460.896484375 disk.rbd1.out‚ 2.146484375 cpu‚25.327809627558004 network.sent‚ 1.177423e+07 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 162.80078125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd1.in‚0 disk.sda3.out‚ 162.80078125 disk.drbd1777.out‚0% proc.memory.availableMB‚ 3746.0625) proc.memory.percent‚16.63668155670166% disk./.usageGB‚1108.6116905212402! disk.rbd2.out‚325.80224609375 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 162.80078125 disk.rbd3.in‚0& disk.nvme0c0n1.out‚41460.896484375# proc.memory.rssMB‚ 2725.75390625eó’«2ñ- _wandbruntime‚4241  _runtime‚4241\„ü÷Èò ' lr-AdamW/pg1‚0.00028263431803672107 ' lr-AdamW/pg2‚0.00028263431803672107 trainer/global_step‚4999 $ _timestamp‚1.788441801536357e+09 _runtime‚4241.581029092 _step‚408ÅwTÍóÇ _step‚408 ' lr-AdamW/pg1‚0.00028263431803672107 ' lr-AdamW/pg2‚0.00028263431803672107 trainer/global_step‚4999 $ _timestamp‚1.788441801536357e+09 _runtime‚4241.581029092˜ùj9t2ô- _wandbruntime‚4241  _runtime‚4241XF¶7ÿ'õùO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477614829920256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549379807691776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 trainer/global_step‚4999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73932839 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚558321258452992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167278214 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9283720642560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9268323213704 _runtime‚ 4241.58209479 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93325335 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567602352494592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚558321232061440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12723247777280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _step‚409 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73932845 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73932839 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93345369 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9281120433152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚567604979095552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34753869 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558648156333448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82734589 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12723292076544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477614801785856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚558321258452992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549379833119744 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34753426 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490338121996800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚567604979095552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34753869 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558648156333448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82734589 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12723292076544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477614801785856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚558321258452992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549379833119744 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34753426 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490338121996800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚558321258452992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167278214 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9283720642560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9268323213704 _runtime‚ 4241.58209479 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93325335 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567602352494592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚558321232061440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12723247777280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074™ý2÷- _wandbruntime‚4242  _runtime‚4242h¯5S 'ø„N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558429198158336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9282952761856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73947081 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93343569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12725775022592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558756326850596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9285552984576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47989916 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490432964861952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167290644 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884418022732706e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚4242.318630969 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9267554271704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚567712124528640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚410 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93343569 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9270152556580 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 trainer/global_step‚4999 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34760542 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558429171766784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12725819308544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚567714751142912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82750020 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167290644 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73947075 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34760107 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558429171766784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73947075 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73947081 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477707217973760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549486148866048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93363629 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558429198158336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93363629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490433037282304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9282952761856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚549486174294016 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93343569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12725775022592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558756326850596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9285552984576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47989916 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490432964861952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167290644 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884418022732706e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚4242.318630969 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9267554271704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚567712124528640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚410 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93343569 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9270152556580 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 trainer/global_step‚4999 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34760542 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558429171766784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12725819308544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚567714751142912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82750020 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167290644 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73947075 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34760107 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558429171766784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73947075 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73947081 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477707217973760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549486148866048 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93363629 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558429198158336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93363629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490433037282304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9282952761856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚549486174294016 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73947081 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400šõLv2ú- _wandbruntime‚4242  _runtime‚4242ƒ ]û× .loss/FBt4/detection_loss‚2.116129994392395 /loss/FRt2/assignment_loss‚2.586676909643061 0loss/FRt3/assignment_loss‚3.6114343961079913 1loss/FBt1/assignment_loss‚0.12995260607522394 epoch‚4 2loss/SRqqt1/assignment_loss‚1.1404895720215371 2loss/SRqqt3/assignment_loss‚2.1170130763892776 %loss/total_loss‚43.13600787335007 2loss/SRqqt4/assignment_loss‚1.4000539753814132 1loss/SRqqt3/detection_loss‚2.6852368160530373 trainer/global_step‚4999 % _timestamp‚1.7884418022754478e+09 _runtime‚4242.320029775 /loss/FRt4/assignment_loss‚2.643589379787445 2loss/FBt2/assignment_loss‚0.004982580120364825 /loss/FRt1/detection_loss‚1.5952245077612448 0loss/FRt1/assignment_loss‚1.3767329563899917 /loss/FRt3/detection_loss‚2.7698206981023152 1loss/SRqqt2/detection_loss‚1.7735081547323395 .loss/FBt2/detection_loss‚3.848926862080892 0loss/FBt3/assignment_loss‚0.1723924332036404 0loss/FBt4/assignment_loss‚0.3410622648273905 /loss/FRt2/detection_loss‚2.9540954048142716 /loss/FRt4/detection_loss‚2.2011783540248873 1loss/SRqqt1/detection_loss‚1.1001743882894517 1loss/SRqqt4/detection_loss‚1.7807071145091737 /loss/FBt1/detection_loss‚0.7537767970448117 .loss/FBt3/detection_loss‚2.498689390718937 2loss/SRqqt2/assignment_loss‚1.5341592408789724 _step‚411@u¬büÜ 0loss/FBt3/assignment_loss‚0.1723924332036404 0loss/FBt4/assignment_loss‚0.3410622648273905 /loss/FRt2/detection_loss‚2.9540954048142716 /loss/FRt4/detection_loss‚2.2011783540248873 1loss/SRqqt1/detection_loss‚1.1001743882894517 1loss/SRqqt4/detection_loss‚1.7807071145091737 /loss/FBt1/detection_loss‚0.7537767970448117 .loss/FBt3/detection_loss‚2.498689390718937 2loss/SRqqt2/assignment_loss‚1.5341592408789724 _step‚411 .loss/FBt4/detection_loss‚2.116129994392395 /loss/FRt2/assignment_loss‚2.586676909643061 0loss/FRt3/assignment_loss‚3.6114343961079913 1loss/FBt1/assignment_loss‚0.12995260607522394 epoch‚4 2loss/SRqqt1/assignment_loss‚1.1404895720215371 2loss/SRqqt3/assignment_loss‚2.1170130763892776 %loss/total_loss‚43.13600787335007 2loss/SRqqt4/assignment_loss‚1.4000539753814132 1loss/SRqqt3/detection_loss‚2.6852368160530373 trainer/global_step‚4999 % _timestamp‚1.7884418022754478e+09 _runtime‚4242.320029775 /loss/FRt4/assignment_loss‚2.643589379787445 2loss/FBt2/assignment_loss‚0.004982580120364825 /loss/FRt1/detection_loss‚1.5952245077612448 0loss/FRt1/assignment_loss‚1.3767329563899917 /loss/FRt3/detection_loss‚2.7698206981023152 1loss/SRqqt2/detection_loss‚1.7735081547323395 .loss/FBt2/detection_loss‚3.848926862080892›>; tžý‚(:“ ÌéåÔ¹®·Ê gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.744' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.54880000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J³¥=þ‚(:² ÌéåÔ’“ÃË disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5625$ memory_percent‚77.13770866394043) proc.memory.percent‚16.63668155670166 disk.rbd2.in‚0& disk.nvme0c0n1.out‚41558.373046875! disk.nvme0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875# proc.memory.rssMB‚ 2725.75390625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1754.04296875% disk./.usageGB‚1108.6116333007812 disk.sda3.in‚0.5625 disk.rbd0.out‚0 cpu‚25.326823747790176 network.sent‚ 1.1779995e+07 disk.rbd1.out‚ 2.146484375! disk.rbd2.out‚329.97412109375 disk.sda.out‚ 163.1640625 disk.rbd3.out‚0) disk./.usagePercent‚74.39985689793254 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚41558.373046875 disk.sda3.out‚ 163.1640625 disk.dm-1.out‚ 163.1640625( proc.memory.availableMB‚ 3745.7578125! disk.dm-0.out‚41558.373046875 network.recv‚ 1.584483e+06T¦•Ëžÿ‚(:“ ÛéåÔž—öÊ gpu.0.gpu‚78.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.634' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.92680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¦z¨7€‚(:¬ ÛéåÔØÒ—Í) disk./.usagePercent‚74.39985689793254 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 163.6953125 disk.sda.out‚ 163.6953125! disk.nvme0n1.in‚ 1754.04296875 disk.rbd2.out‚ 335.423828125# disk.nvme0n1.out‚41558.81640625 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-0.out‚41558.81640625$ memory_percent‚77.13685035705566 cpu‚25.325834805681897 network.sent‚ 1.188933e+07 network.recv‚ 1.591491e+06 disk.sda1.in‚0 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.out‚ 163.6953125# disk.nvme0c0n1.in‚ 1754.04296875% disk.nvme0c0n1.out‚41558.81640625( proc.memory.availableMB‚ 3745.8984375# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166% disk./.usageGB‚1108.6116333007812 disk.rbd1.out‚ 2.146484375 disk.rbd3.out‚0 disk.dm-1.in‚0.5625 proc.cpu.threads‚17áÁÓ4’‚(:‡ êéåÔ✐ gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚113.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0![YU‚‚(:­ êéåÔ™‡¸Ë disk.sda3.in‚0.5625% disk.nvme0c0n1.out‚41559.11328125 disk.sda.out‚ 163.97265625 proc.cpu.threads‚17) disk./.usagePercent‚74.39980876851611 disk.sda3.out‚ 163.97265625% disk./.usageGB‚1108.6109161376953# disk.nvme0n1.out‚41559.11328125 disk.dm-0.in‚ 1754.04296875$ memory_percent‚89.08100128173828 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚41559.11328125 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.rbd2.in‚0) proc.memory.percent‚16.63668155670166 network.sent‚ 1.1894653e+07 network.recv‚ 1.595665e+06 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-1.out‚ 163.97265625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rb ‚ãd0.out‚0 disk.sda.in‚0.5625 disk.sda1.out‚0& proc.memory.availableMB‚ 1788.96875# proc.memory.rssMB‚ 2725.75390625 cpu‚25.32489437711209 disk.rbd2.out‚ 335.423828125! disk.nvme0n1.in‚ 1754.04296875‡Û2ƒ- _wandbruntime‚4281  _runtime‚4281˜ì[Ç„Á & lr-AdamW/pg1‚0.0002801396684778799 & lr-AdamW/pg2‚0.0002801396684778799 trainer/global_step‚5049 % _timestamp‚1.7884418409785879e+09 _runtime‚4281.023518173 _step‚412ñmhÌ…Æ _runtime‚4281.023518173 _step‚412 & lr-AdamW/pg1‚0.0002801396684778799 & lr-AdamW/pg2‚0.0002801396684778799 trainer/global_step‚5049 % _timestamp‚1.7884418409785879e+09œpÑ2†- _wandbruntime‚4281  _runtime‚4281¥zY(‡úO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 % _timestamp‚1.7884418409793167e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573090968610816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12849897716736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564054072059020 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9357205653644 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563718243720192 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94240387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74644645 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94240387 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573093595211776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚554696891833344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚563718217328640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74644645 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9372751282176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚5049 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495084102177792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94260421 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83531891 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74644639 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563718217328640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9375351491584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573090968610816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495084174611456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482234232595456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9375351491584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35090932 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94260421 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48440520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564056695771656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74644639 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12849942016000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9372751282176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573093595211776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482234204461056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83531449 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚554696866405376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9359803938312 _step‚413 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168885026 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563718243720192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 _runtime‚4281.024451893QY (ˆÿO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚554696891833344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚563718217328640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74644645 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9372751282176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚5049 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495084102177792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94260421 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83531891 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74644639 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚563718217328640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9375351491584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573090968610816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495084174611456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482234232595456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9375351491584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35090932 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94260421 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48440520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564056695771656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74644639 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12849942016000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9372751282176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573093595211776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482234204461056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83531449 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚554696866405376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9359803938312 _step‚413 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168885026 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚563718243720192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 _runtime‚4281.024451893 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 % _timestamp‚1.7884418409793167e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573090968610816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12849897716736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564054072059020 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9357205653644 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚563718243720192 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94240387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74644645 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94240387 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573093595211776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12852454305792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚554803207579648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35097624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564162242575960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚563826157033984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573200740644864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9359034996312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94258621 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9374583610880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74658875 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573203367259136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9374583610880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚563826183425536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573203367259136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 trainer/global_step‚5049 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48449716 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74658881 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83547337 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12852412116992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48449713 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94278681 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9361633281188 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94278681 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 % _timestamp‚1.7884418417178006e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495179004631552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚563826183425536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168917496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9377183833600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83547774 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482326592514560 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573203367259136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 trainer/global_step‚»Ýäjp!5049 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48449716 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74658881 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83547337 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12852412116992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48449713 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94278681 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9361633281188 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94278681 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 % _timestamp‚1.7884418417178006e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495179004631552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚563826183425536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168917496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9377183833600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83547774 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482326592514560 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12852454305792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚554803207579648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35097624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564162242575960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚563826157033984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573200740644864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9359034996312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94258621 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0žbºVµ2Œ- _wandbruntime‚4281  _runtime‚4281Ä¢SÍ 1loss/FBt4/assignment_loss‚0.26753761777136886 /loss/SRqqt3/detection_loss‚2.40439006013255 /loss/FBt3/detection_loss‚2.2277205189069114 .loss/FRt1/detection_loss‚1.631287407263049 &loss/total_loss‚40.142744579708996 $ _timestamp‚1.788441841719491e+09 0loss/FBt2/assignment_loss‚0.2938166251963897 trainer/global_step‚5049 0loss/FRt4/assignment_loss‚2.8193555380168713 0loss/FBt1/assignment_loss‚0.1386731386349078 1loss/FBt3/assignment_loss‚0.10500652956155439 _step‚415 .loss/FRt3/detection_loss‚2.286704922865527 epoch‚4 2loss/SRqqt3/assignment_loss‚1.9690941514507416 .loss/FRt2/detection_loss‚2.469712802143984 /loss/FRt4/detection_loss‚2.5927503148191855 0loss/FRt1/assignment_loss‚1.1211268327065878 /loss/FRt2/assignment_loss‚2.760505185570828 1loss/SRqqt1/detection_loss‚0.8844250993452211 1loss/SRqqt4/detection_loss‚2.0392900786616583 .loss/FBt1/detection_loss‚0.827106795726797 .loss/FBt4/detection_loss‚2.052203502919939 0loss/FRt3/assignment_loss‚3.0356276916420977 /loss/SRqqt2/detection_loss‚1.92023096320241 /loss/FBt2/detection_loss‚2.5587491393089294 _runtime‚4281.764243558 1loss/SRqqt1/assignment_loss‚0.774017730511833 2loss/SRqqt2/assignment_loss‚1.3815338476019543 /loss/SRqqt4/assignment_loss‚1.5818780857477{o¿XŽÒ 0loss/FBt2/assignment_loss‚0.2938166251963897 trainer/global_step‚5049 0loss/FRt4/assignment_loss‚2.8193555380168713 0loss/FBt1/assignment_loss‚0.1386731386349078 1loss/FBt3/assignment_loss‚0.10500652956155439 _step‚415 .loss/FRt3/detection_loss‚2.286704922865527 epoch‚4 2loss/SRqqt3/assignment_loss‚1.9690941514507416 .loss/FRt2/detection_loss‚2.469712802143984 /loss/FRt4/detection_loss‚2.5927503148191855 0loss/FRt1/assignment_loss‚1.1211268327065878 /loss/FRt2/assignment_loss‚2.760505185570828 1loss/SRqqt1/detection_loss‚0.8844250993452211 1loss/SRqqt4/detection_loss‚2.0392900786616583 .loss/FBt1/detection_loss‚0.827106795726797 .loss/FBt4/detection_loss‚2.052203502919939 0loss/FRt3/assignment_loss‚3.0356276916420977 /loss/SRqqt2/detection_loss‚1.92023096320241 /loss/FBt2/detection_loss‚2.5587491393089294 _runtime‚4281.764243558 1loss/SRqqt1/assignment_loss‚0.774017730511833 2loss/SRqqt2/assignment_loss‚1.3815338476019543 /loss/SRqqt4/assignment_loss‚1.5818780857477 1loss/FBt4/assignment_loss‚0.26753761777136886 /loss/SRqqt3/detection_loss‚2.40439006013255 /loss/FBt3/detection_loss‚2.2277205189069114 .loss/FRt1/detection_loss‚1.631287407263049 &loss/total_loss‚40.142744579708996 $ _timestamp‚1.788441841719491e+09Ÿ磒”‚(:‰ ùéåÔ¡þÁÉ gpu.0.gpu‚72.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚162.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©w8†4‚(:© ùéåԈˡË# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 disk.sda.in‚0.5625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚41657.240234375 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.sent‚ 1.1900687e+07 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.rbd2.out‚ 336.111328125 disk.sda.out‚ 164.65625 disk.sda2.in‚0$ disk.nvme0n1.out‚41657.240234375 disk.sda3.out‚ 164.65625! disk.dm-0.out‚41657.240234375# memory_percent‚89.0810489654541 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 cpu‚25.323883498708096 network.recv‚ 1.600281e+06) disk./.usagePercent‚74.39994368448664 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd2.in‚0( proc.memory.availableMB‚ 1788.9609375% disk./.usageGB‚1108.6129264831543 disk.dm-1.out‚ 164.65625 disk.rbd3.in‚0 disk.rbd1.out‚ 2.146484375ïWF”‘‚(:‰ ˆêåÔ–ØàÈ gpu.0.gpu‚72.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚239.508' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.9016 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x´¯`5’‚(:ª ˆêåÔ¹œéË disk.sda3.in‚0.5625 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 165.015625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2725.75390625) disk./.usagePercent‚74.39993651627569 disk.sda.out‚ 165.015625 disk.sda1.out‚0 disk.rbd0.out‚0# memory_percent‚89.0829086303711& proc.memory.availableMB‚ 1788.65625 cpu‚25.322986283634897$ disk.nvme0n1.out‚41657.583984375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚41657.583984375 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 165.015625 disk.rbd2.out‚ 336.134765625 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1754.04296875! disk.dm-0.out‚41657.583984375 disk.rbd1.out‚ 2.146484375) proc.memory.percent‚16.63668155670166 network.recv‚ 1.606849e+06% disk./.usageGB‚1108.6128196716309 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0 network.sent‚ 1.2009272e+07 disk.drbd1777.in‚0 disk.dm-1.in‚0.5625 proc.cpu.threads‚17§½*”“‚(:‰ —êåÔËà¤É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚131.079' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ààt5”‚(:ª —êåÔ†¬ñË! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.out‚ 41755.671875% disk./.usageGB‚1108.6128768920898 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 165.61328125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.out‚ 41755.671875( proc.memory.availableMB‚ 1788.6640625 network.recv‚ 1.611264e+06 disk.rbd1.out‚ 2.146484375 proc.cpu.threads‚17 disk.rbd2.out‚ 336.486328125 disk.sda.in‚0.5625 disk.sda2.out‚0! disk.nvme0n1.out‚ 41755.671875 disk.dm-0.in‚ 1754.04296875$ memory_percent‚89.08286094665527# proc.memory.rssMB‚ 2725.75390625 cpu‚25.322043206637783 disk.sda1.in‚0 disk.rbd3.in‚0) proc.memory.percent‚16.63668155670166) disk./.usagePercent‚74.39994035638871 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.dm-1.out‚ 165.61328125 network.sent‚ 1.2015538e+07 disk.sda.out‚ 165.61328125 disk.dm-1.in‚0.5625´Á<2•- _wandbruntime‚4320  _runtime‚4320ôw[‹É–à trainer/global_step‚5099 % _timestamp‚1.7884418799864194e+09 _runtime‚4320.031012544 _step‚416 ' lr-AdamW/pg1‚0.00027762701691020177 ' lr-AdamW/pg2‚0.00027762701691020177«{Ò³Î—È _step‚416 ' lr-AdamW/pg1‚0.00027762701691020177 ' lr-AdamW/pg2‚0.00027762701691020177 trainer/global_step‚5099 % _timestamp‚1.7884418799864194e+09 _runtime‚4320.031012544 65IB2˜- _wandbruntime‚4320  _runtime‚43201°f(™úO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚560013925118976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569462611497228 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9448686378252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 % _timestamp‚1.7884418799870892e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚578579584727040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486853635270656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚417 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚578579584727040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35430965 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95175473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚569115202595840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48900317 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95155439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚569115228987392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95175473 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12976343120896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚499830022690816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9464382131200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75356439 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚578582211328000 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _runtime‚4320.031973234 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48900320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 trainer/global_step‚5099 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12976387420160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ýàœ‘îšÿO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12976387420160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚560013925118976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569462611497228 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9448686378252 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 % _timestamp‚1.7884418799870892e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚578579584727040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486853635270656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚417 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚578579584727040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35430965 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95175473 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚569115202595840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48900317 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95155439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚569115228987392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95175473 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12976343120896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚499830022690816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9464382131200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75356439 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚578582211328000 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _runtime‚4320.031973234 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48900320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 trainer/global_step‚5099 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968¡BBÐ2›- _wandbruntime‚4320  _runtime‚4320ߟ{ 'œ„N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75370681 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95193733 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499924948692992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569570782014168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75370681 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9450515720920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9466214459904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569223168692736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486945995189760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84346712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499924874175488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12978925368832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _step‚418 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75370675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578691983375360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569223142301184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚560120266293248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75370675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48909516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35437651 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 % _timestamp‚1.7884418807339277e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9468814682624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9466214459904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9468814682624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569223168692736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35437199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚560120291721216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 trainer/global_step‚5099 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9453114005796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84347167~º)r'‰N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569223168692736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35437199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚560120291721216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 trainer/global_step‚5099 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9453114005796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84347167 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75370681 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95193733 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499924948692992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569570782014168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75370681 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9450515720920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9466214459904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569223168692736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486945995189760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84346712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499924874175488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12978925368832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _step‚418 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75370675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578691983375360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569223142301184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚560120266293248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75370675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48909516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35437651 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 % _timestamp‚1.7884418807339277e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9468814682624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9466214459904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9468814682624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 F±ƒ}ɦà _runtime‚4359.884460269 _step‚420 ' lr-AdamW/pg1‚0.00027509688696650703 ' lr-AdamW/pg2‚0.00027509688696650703 trainer/global_step‚5149 % _timestamp‚1.7884419198398137e+09Ù™›XÎ§È _runtime‚4359.884460269 _step‚420 ' lr-AdamW/pg1‚0.00027509688696650703 ' lr-AdamW/pg2‚0.00027509688696650703 trainer/global_step‚5149 % _timestamp‚1.7884419198398137e+09¤?>—2¨- _wandbruntime‚4359  _runtime‚4359©‘N©øO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35769828 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76068245 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584068200843264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9558613189632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76068239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49360117 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚574871150935436 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9556012980224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚574512214254592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85129491 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76068239 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49360117 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚574871150935436 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9556012980224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚574512214254592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85129491 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584068200843264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9558613189632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76082481 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9560445531648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚565437325006848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85145346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85145783 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13105769853952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35776467 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584180599491584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504671195853312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76082475 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504671123432960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9541996445528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9560445531648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚574620153959936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13105725568000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96088725 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96108785 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9557845308928 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9544594730404 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 $ _timestamp‚1.788441920591729e+09 _step‚422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35776033 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574620127568384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96108785 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584180599491584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚565437350434816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584177972877312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚491565425999360˜ÎBŸ.­ˆN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574620127568384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96108785 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584180599491584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚565437350434816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584177972877312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚491565425999360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76082481 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9560445531648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚565437325006848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85145346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85145783 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13105769853952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35776467 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584180599491584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504671195853312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76082475 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504671123432960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9541996445528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9560445531648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚574620153959936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13105725568000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96088725 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96108785 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9557845308928 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9544594730404 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 $ _timestamp‚1.788441920591729e+09 _step‚422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35776033 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¦§x_Ó2®- _wandbruntime‚4360  _runtime‚4360·vp<[¯Õ -loss/FBt2/detection_loss‚2.57037572728263 0loss/FBt3/assignment_loss‚0.0647574806753255 /loss/FBt3/detection_loss‚2.5921725928783417 %loss/total_loss‚43.24469406129241 epoch‚4 trainer/global_step‚5149 2loss/SRqqt3/assignment_loss‚1.9284195254246395 2loss/SRqqt4/assignment_loss‚1.3861524224154314 1loss/FBt2/assignment_loss‚0.06420996672221059 /loss/FRt1/detection_loss‚2.1025664263257853 /loss/FBt1/detection_loss‚0.6459047284897887 1loss/SRqqt2/assignment_loss‚1.334681008315208 1loss/SRqqt1/detection_loss‚1.0041029866446147 1loss/SRqqt2/detection_loss‚1.9534858939599018 0loss/SRqqt3/detection_loss‚2.137054381105635 /loss/FBt4/detection_loss‚3.2632136940956116 _runtime‚4360.638364554 1loss/FBt1/assignment_loss‚0.07060104999471488 .loss/FRt3/detection_loss‚2.896894701065556 /loss/FRt2/assignment_loss‚3.056482020096901 /loss/FRt4/detection_loss‚2.4156483800514885 _step‚423 % _timestamp‚1.7884419205937676e+09 2loss/SRqqt1/assignment_loss‚0.9024971216832373 0loss/FBt4/assignment_loss‚0.0267552696459461 1loss/SRqqt4/detection_loss‚1.9919627830386162 0loss/FRt3/assignment_loss‚3.2611152491261883 0loss/FRt4/assignment_loss‚3.7497331487095873 /loss/FRt2/detection_loss‚2.5723077265115886 0loss/FRt1/assignment_loss‚1.2535997770334546„ɵ*`°Ú epoch‚4 trainer/global_step‚5149 2loss/SRqqt3/assignment_loss‚1.9284195254246395 2loss/SRqqt4/assignment_loss‚1.3861524224154314 1loss/FBt2/assignment_loss‚0.06420996672221059 /loss/FRt1/detection_loss‚2.1025664263257853 /loss/FBt1/detection_loss‚0.6459047284897887 1loss/SRqqt2/assignment_loss‚1.334681008315208 1loss/SRqqt1/detection_loss‚1.0041029866446147 1loss/SRqqt2/detection_loss‚1.9534858939599018 0loss/SRqqt3/detection_loss‚2.137054381105635 /loss/FBt4/detection_loss‚3.2632136940956116 _runtime‚4360.638364554 1loss/FBt1/assignment_loss‚0.07060104999471488 .loss/FRt3/detection_loss‚2.896894701065556 /loss/FRt2/assignment_loss‚3.056482020096901 /loss/FRt4/detection_loss‚2.4156483800514885 _step‚423 % _timestamp‚1.7884419205937676e+09 2loss/SRqqt1/assignment_loss‚0.9024971216832373 0loss/FBt4/assignment_loss‚0.0267552696459461 1loss/SRqqt4/detection_loss‚1.9919627830386162 0loss/FRt3/assignment_loss‚3.2611152491261883 0loss/FRt4/assignment_loss‚3.7497331487095873 /loss/FRt2/detection_loss‚2.5723077265115886 0loss/FRt1/assignment_loss‚1.2535997770334546 -loss/FBt2/detection_loss‚2.57037572728263 0loss/FBt3/assignment_loss‚0.0647574806753255 /loss/FBt3/detection_loss‚2.5921725928783417 %loss/total_loss‚43.24469406129241§ jêßœ±‚(:‘ ÄêåÔ‹ò‡É gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚261.068' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.21359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡·Á¢D²‚(:¹ ÄêåÔÓ¼–Ë disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚336.6083984375 disk.sda.in‚0.5625& disk.nvme0n1.out‚41854.01513671875 disk.sda3.in‚0.5625) disk./.usagePercent‚74.37105553831545$ disk./.usageGB‚1108.182472229004 disk.dm-1.out‚ 167.05078125 disk.sda3.out‚ 167.05078125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚41854.01513671875 network.sent‚ 1.2135448e+07 disk.rbd2.in‚0 disk.sda.out‚ 167.05078125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1754.04296875 proc.cpu.threads‚17# disk.dm-0.out‚41854.01513671875 disk.drbd1777.in‚0 disk.rbd3.in‚0$ memory_percent‚89.08250331878662 network.recv‚ 1.625851e+06 disk.dm-1.in‚0.5625 disk.rbd3.out‚0 cpu‚25.31920744170477 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 disk.dm-0.in‚ 1754.04296875 disk.sda1.out‚0) proc.memory.availableMB‚ 1788.72265625{uWž³‚(:“ ÓêåÔ‚ïÝÈ gpu.0.gpu‚52.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚276.319' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.26380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hñ™“B´‚(:· ÓêåÔÂÈœÌ) disk./.usagePercent‚74.37105937842846 disk.rbd2.out‚336.6708984375 disk.sda.out‚ 167.55078125 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 167.55078125 cpu‚25.31833543131776 disk.rbd1.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚41854.31591796875# disk.dm-0.out‚41854.31591796875$ memory_percent‚89.08360004425049$ disk./.usageGB‚1108.182529449463 disk.sda1.in‚0 disk.sda3.out‚ 167.55078125 proc.cpu.threads‚17 network.sent‚ 1.22435e+07 disk.sda.in‚0.5625 disk.rbd3.in‚0# proc.memory.rssMB‚ 2725.75390625 network.recv‚ 1.631813e+06 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚41854.31591796875) proc.memory.availableMB‚ 1788.54296875) proc.memory.percent‚16.63668155670166 disk.sda2.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875:×Øžµ‚(:“ âêåÔþ’ëÈ gpu.0.gpu‚67.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.829' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.36580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÖ D¶‚(:¹ âêåÔÿ­Ì network.recv‚ 1.63571e+06) disk./.usagePercent‚74.37105553831545$ disk./.usageGB‚1108.182472229004 disk.rbd1.out‚ 2.146484375 disk.sda.in‚0.5625 disk.dm-0.in‚ 1754.04296875$ memory_percent‚89.08202648162842) proc.memory.percent‚16.63668155670166 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-1.in‚0.5625 cpu‚25.317427668925145 disk.rbd2.out‚336.6708984375( disk.nvme0c0n1.out‚41952.07373046875 disk.rbd3.in‚0 disk.sda3.in‚0.5625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 network.sent‚ 1.2248769e+07 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 168.10546875! disk.nvme0n1.in‚ 1754.04296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1788.80078125# proc.memory.rssMB‚ 2725.75390625& disk.nvme0n1.out‚41952.07373046875 disk.sda3.out‚ 168.10546875# disk.dm-0.out‚41952.07373046875 disk.dm-1.out‚ 168.10546875Rl®¥2·- _wandbruntime‚4400  _runtime‚4400 }ÀǸÁ & lr-AdamW/pg1‚0.0002725498059220839 & lr-AdamW/pg2‚0.0002725498059220839 trainer/global_step‚5199 % _timestamp‚1.7884419600450275e+09 _runtime‚4400.089687931 _step‚424 ³2Ì¹Æ _runtime‚4400.089687931 _step‚424 & lr-AdamW/pg1‚0.0002725498059220839 & lr-AdamW/pg2‚0.0002725498059220839 trainer/global_step‚5199 % _timestamp‚1.7884419600450275e+09¨ИâÛ2º- _wandbruntime‚4400  _runtime‚4400À(»úO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76780039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49819917 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579909173130240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96985543 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9650244038656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9650244038656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4400.090768547 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579909199521792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9647643829248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚509322219586560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9647643829248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96985543 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85927248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496092440621056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚570648042546176 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49819920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509322292020224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36107775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚589559443560448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _step‚425 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579909173130240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580279690373644 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9631647827468 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97005577 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85927695 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579909199521792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97005577 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚570648067974144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36107331 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13229851399168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580282314086280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚589556816959488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173785622 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚589556816959488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76780045 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9634246112136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574¶q'š*#¼ÿO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36107775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚589559443560448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _step‚425 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579909173130240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580279690373644 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9631647827468 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97005577 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85927695 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579909199521792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97005577 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚570648067974144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36107331 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13229851399168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580282314086280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚589556816959488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173785622 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚589556816959488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76780045 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9634246112136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76780039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49819917 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579909173130240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96985543 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9650244038656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9650244038656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4400.090768547 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579909199521792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9647643829248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚509322219586560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9647643829248«ÂxBÛ @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96985543 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85927248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496092440621056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚570648042546176 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49819920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509322292020224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672©+YˆÁ2½- _wandbruntime‚4400  _runtime‚4400.·í '¾ƒN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85943153 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97023837 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49829113 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚580017139227136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509417124966912 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _runtime‚ 4400.84237251 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76794281 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97003777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509417197387264 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9633477170136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚423 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884419607968981e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9649476157952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚589669215607808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚570754409148416 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76794281 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85943576 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76794275 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97023837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36114040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9649476157952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚589669215607808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496184828674560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚570754383720448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0B •'¿ˆN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97003777 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚580017112835584 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9633477170136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚423 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884419607968981e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9649476157952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚589669215607808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚570754409148416 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76794281 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85943576 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76794275 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97023837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36114040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9649476157952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚589669215607808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496184828674560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚570754383720448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85943153 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97023837 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49829113 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚580017139227136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509417124966912 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _runtime‚ 4400.84237251 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76794281 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97003777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509417197387264 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77491839 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36445285 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9739274678272 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚595048059676672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86725479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚5249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500711843296256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585690853524488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575965126687744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595045433075712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514068023561728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 _runtime‚4439.857526323 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚585306184788992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚474 GADeviceStatsMonitor.on_trainAæìú_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36445759 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514067949030912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9725726836744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9723128552076 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚585306184788992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50279720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500711815161856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9741874887680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585688229811852 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77491845 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9741874887680 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585688229811852 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77491845 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77491839 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36445285 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9739274678272 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚595048059676672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86725479 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚5249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500711843296256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585690853524488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575965126687744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595045433075712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514068023561728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 _runtime‚4439.857526323 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚585306184788992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36445759 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514067949030912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9725726836744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9723128552076 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚585306184788992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50279720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248­úÍO’Í‚(:‡ ëåÔ•ø©Ê gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚257.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È“D΂(:¹ ëåÔ©¯žË disk.rbd2.in‚0( disk.nvme0c0n1.out‚42050.39111328125 cpu‚25.31479857710129 network.recv‚ 1.651558e+06 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.sda2.in‚0 disk.sda3.in‚0.5625 network.sent‚ 1.2369668e+07 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.dm-1.in‚0.5625# proc.memory.rssMB‚ 2725.75390625 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 169.48828125 disk.rbd3.out‚0% disk./.usageGB‚1108.1809425354004& disk.nvme0n1.out‚42050.39111328125 disk.sda3.out‚ 169.48828125 disk.rbd0.out‚0 disk.dm-0.in‚ 1754.04296875# disk.dm-0.out‚42050.39111328125# memory_percent‚89.0852689743042) proc.memory.percent‚16.63668155670166) disk./.usagePercent‚74.37095287929421 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.rbd2.out‚336.7998046875) proc.memory.availableMB‚ 1788.26953125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 169.48828125Ù2Ï- _wandbruntime‚4440  _runtime‚4440¡§ül 'ЄN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514162923476480 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9741107006976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50288913 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36452447 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13358692126720 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4440.615778294 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97918829 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9741107006976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77506081 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9743707229696 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50288916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚595157831724032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595155205109760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36451996 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585796400328792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9727556179620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77506081 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86740909 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97938889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595157831724032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77506075 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97938889 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚585414124494336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9724957894744 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚576071442434048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚585414124494336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175424904ýLR Ñ‰N % _timestamp‚1.7884420005703123e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9741107006976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77506081 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9743707229696 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50288916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚595157831724032 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595155205109760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36451996 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585796400328792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9727556179620 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77506081 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚430 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86740909 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocatior–©:ons.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97938889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595157831724032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77506075 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97938889 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚585414124494336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9724957894744 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚576071442434048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚585414124494336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175424904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514162923476480 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9741107006976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50288913 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36452447 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13358692126720 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4440.615778294 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97918829 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600534049191936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9830905527296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 _step‚433 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177039326 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9817207561352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522063 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177019286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78203645 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518813295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590703170056192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98835681 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521607 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9814609276684 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590703143664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50739517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505331245971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13482124440576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 trainer/global_step‚5299 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚581282185401344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50739520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚591099392962696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590703143664640øreÞ(ÞÿO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9814609276684 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590703143664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50739517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505331245971456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13482124440576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 trainer/global_step‚5299 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚581282185401344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50739520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚591099392962696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590703143664640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚581282159973376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9833505736704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78203639 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78203639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚600536675792896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚600536675792896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9833505736704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591096769250060 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600534049191936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9830905527296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 _step‚433 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177039326 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9817207561352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522063 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177019286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78203645 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518813295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚590703170056192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98835681 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521607 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552±¾G 2ß- _wandbruntime‚4480  _runtime‚4480ê;©ùZà„N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505423605890560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9832737856000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13484610212352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78217875 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87537960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98853941 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9835338078720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590811083369984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98833881 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591207563479844 % _timestamp‚1.7884420403652952e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600643821225984 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98853941 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36789244 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591204939767000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177071822 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98833881 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78217881 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9832737856000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78217881 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87537524 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚590811083369984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9816438619352 trainer/global_step‚5299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78217875 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚600646447840256 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581388501147648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600646447840256 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5813885265756164c—'á‰N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚581388526575616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505423605890560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9832737856000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13484610212352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78217875 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87537960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98853941 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9835338078720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590811083369984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98833881 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591207563479844 % _timestamp‚1.7884420403652952e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600643821225984 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98853941 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36789244 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591204939767000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177071822 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98833881 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78217881 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9832737856000 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78217881 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87537524 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚590811083369984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9816438619352 trainer/global_step‚5299 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78217875 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚600646447840256 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581388501147648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600646447840256 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0²úÑý2â- _wandbruntime‚4480  _runtime‚4480¦ÍlXãÒ 0loss/FBt4/assignment_loss‚1.1475160792469978 2loss/SRqqt1/assignment_loss‚0.9847536668250406 -loss/FRt4/detection_loss‚2.63961573146485 epoch‚4 trainer/global_step‚5299 % _timestamp‚1.7884420403668365e+09 /loss/FRt2/assignment_loss‚2.755959930808045 /loss/FBt1/detection_loss‚0.5094153347308747 /loss/FBt2/detection_loss‚2.0501586523923008 2loss/SRqqt2/assignment_loss‚1.6783380722006163 1loss/SRqqt4/assignment_loss‚1.498868924965791 0loss/FBt1/assignment_loss‚0.2254611643729591 /loss/FRt3/detection_loss‚2.4212153921352964 _runtime‚ 4480.41137344 1loss/SRqqt2/detection_loss‚1.9953637295299107 .loss/FBt4/detection_loss‚5.058083713054657 %loss/total_loss‚43.81370930573567 _step‚435 0loss/FRt3/assignment_loss‚3.6278586656816545 1loss/SRqqt3/assignment_loss‚1.540033274683459 .loss/FRt1/detection_loss‚1.755752345705603 /loss/FRt2/detection_loss‚2.3364647904107736 1loss/SRqqt3/detection_loss‚2.4158402249730866 .loss/FBt3/detection_loss‚2.246315394129072 1loss/SRqqt1/detection_loss‚1.2104195591614202 0loss/FRt1/assignment_loss‚1.2256757363359978 0loss/FRt4/assignment_loss‚2.6432839052097217 1loss/FBt2/assignment_loss‚0.11480467798272995 1loss/FBt3/assignment_loss‚0.34525956798877033 1loss/SRqqt4/detection_loss‚1.3872507717460394ý_]ä× 1loss/SRqqt2/detection_loss‚1.9953637295299107 .loss/FBt4/detection_loss‚5.058083713054657 %loss/total_loss‚43.81370930573567 _step‚435 0loss/FRt3/assignment_loss‚3.6278586656816545 1loss/SRqqt3/assignment_loss‚1.540033274683459 .loss/FRt1/detection_loss‚1.755752345705603 /loss/FRt2/detection_loss‚2.3364647904107736 1loss/SRqqt3/detection_loss‚2.4158402249730866 .loss/FBt3/detection_loss‚2.246315394129072 1loss/SRqqt1/detection_loss‚1.2104195591614202 0loss/FRt1/assignment_loss‚1.2256757363359978 0loss/FRt4/assignment_loss‚2.6432839052097217 1loss/FBt2/assignment_loss‚0.11480467798272995 1loss/FBt3/assignment_loss‚0.34525956798877033 1loss/SRqqt4/detection_loss‚1.3872507717460394 0loss/FBt4/assignment_loss‚1.1475160792469978 2loss/SRqqt1/assignment_loss‚0.9847536668250406 -loss/FRt4/detection_loss‚2.63961573146485 epoch‚4 trainer/global_step‚5299 % _timestamp‚1.7884420403668365e+09 /loss/FRt2/assignment_loss‚2.755959930808045 /loss/FBt1/detection_loss‚0.5094153347308747 /loss/FBt2/detection_loss‚2.0501586523923008 2loss/SRqqt2/assignment_loss‚1.6783380722006163 1loss/SRqqt4/assignment_loss‚1.498868924965791 0loss/FBt1/assignment_loss‚0.2254611643729591 /loss/FRt3/detection_loss‚2.4212153921352964 _runtime‚ 4480.41137344³mU—”å‚(:‰ ¼ëåÔµöùÈ gpu.0.gpu‚82.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚263.581' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹î2æ‚(:§ ¼ëåÔ¥Ó¢Ë# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 network.recv‚ 1.667957e+06% disk./.usageGB‚1108.1805572509766 disk.dm-1.out‚ 171.03125 disk.drbd1777.out‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚42149.513671875 network.sent‚ 1.2570238e+07 disk.rbd2.out‚342.3310546875 proc.cpu.threads‚17 disk.sda3.out‚ 171.03125# disk.nvme0c0n1.in‚ 1754.04296875! disk.dm-0.out‚42149.513671875 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1754.04296875$ proc.memory.availableMB‚1788.375 disk.sda3.in‚0.5625 disk.rbd1.out‚ 2.146484375 disk.sda2.out‚0$ memory_percent‚89.08462524414062 cpu‚25.312157154978394 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 171.03125) disk./.usagePercent‚74.37092702253325& disk.nvme0c0n1.out‚42149.513671875 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd2.in‚0©l\”ç‚(:‰ ËëåÔ“´»É gpu.0.gpu‚78.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚207.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êµ8Dè‚(:¹ ËëåÔ»¨¹Ë disk.rbd3.in‚0 disk.sda3.out‚ 171.64453125 disk.rbd0.in‚0 disk.rbd2.out‚347.7138671875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 2.146484375 network.sent‚ 1.2679385e+07 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.rbd2.in‚0) proc.memory.percent‚16.63668155670166 disk.sda.out‚ 171.64453125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚42250.41064453125# disk.dm-0.out‚42250.41064453125) disk./.usagePercent‚74.37113464464352 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.dm-1.out‚ 171.64453125$ memory_percent‚89.08369541168213 disk.rbd1.in‚0) proc.memory.availableMB‚ 1788.52734375$ disk./.usageGB‚1108.183650970459 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.04296875& disk.nvme0n1.out‚42250.41064453125# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875# proc.memory.rssMB‚ 2725.75390625 cpu‚25.31128647053606 network.recv‚ 1.674304e+06¬àˆžé‚(:“ ÚëåÔò×—É gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚260.624' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.12480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=èÔDê‚(:¹ ÚëåÔ„¹¥Ë disk.sda3.in‚0.5625# disk.dm-0.out‚42251.57861328125 disk.rbd2.out‚353.8466796875 disk.sda2.out‚0 disk.sda3.out‚ 172.07421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 172.07421875# disk.nvme0c0n1.in‚ 1754.04296875( disk.nvme0c0n1.out‚42251.57861328125% disk./.usageGB‚1108.1836967468262 disk.rbd3.out‚0 disk.dm-1.in‚0.5625 disk.sda1.out‚0 cpu‚25.310425293010912 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda2.in‚0' proc.memory.availableMB‚ 1788.546875# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 network.sent‚ 1.2684586e+07) disk./.usagePercent‚74.37113771673393& disk.nvme0n1.out‚42251.57861328125 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 172.07421875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1754.04296875$ memory_percent‚89.08357620239258 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd0.out‚0 network.recv‚ 1.677941e+06ñù"Ò2ë- _wandbruntime‚4519  _runtime‚4519šŸ×UÈì _step‚436 ' lr-AdamW/pg1‚0.00026481218126165305 ' lr-AdamW/pg2‚0.00026481218126165305 trainer/global_step‚5349 $ _timestamp‚1.788442079405043e+09 _runtime‚4519.449711514|/ÖÍíÇ _step‚436 ' lr-AdamW/pg1‚0.00026481218126165305 ' lr-AdamW/pg2‚0.00026481218126165305 trainer/global_step‚5349 $ _timestamp‚1.788442079405043e+09 _runtime‚4519.449711514´m`‰ 2î- _wandbruntime‚4519  _runtime‚4519ÌKe›qïúO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9925136585728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51199317 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596100128931840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚606025291909120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596100128931840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596100155323392 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37119402 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9908688285960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51199320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523558967050240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9925136585728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9922536376320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509950620512256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78915439 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596100155323392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚596505308688268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.la¡zþ rge_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13608318403584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884420794058182e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99730699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚606025291909120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523558892519424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509950648646656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78915445 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚606022665308160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9906090001292 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 _step‚437 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99750733 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚596507932400904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99750733 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586599244114944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78915445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37119858 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408—ÍCF(ðÿO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596100155323392 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37119402 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9908688285960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51199320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523558967050240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9925136585728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9922536376320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509950620512256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78915439 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596100155323392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚596505308688268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13608318403584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884420794058182e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99730699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚606025291909120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523558892519424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509950648646656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78915445 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚606022665308160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9906090001292 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 _step‚437 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99750733 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚596507932400904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99750733 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586599244114944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78915445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37119858 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9925136585728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51199317 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596100128931840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚606025291909120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596100128931840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124µ…”èÿ2ñ- _wandbruntime‚4520  _runtime‚4520îµß 'òƒN YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9926968927744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596208068637184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9924368705024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚523653890022912 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88334616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _step‚438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9924368705024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9926968927744 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13610853322752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586705585289216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚606135063956480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚606132437342208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88335041 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9910517628836 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78929675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596208095028736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510043008565760 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78929675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99768993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 $ _timestamp‚1.788442080147941e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51208516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37126103 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78929681 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510043036700160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37126525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4520.193054998 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88334616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _step‚438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9924368705024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9926968927744 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13610853322752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586705585289216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚606135063956480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚606132437342208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88335041 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9910517628836 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78929675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596208095028736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510043008565760 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78929675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99768993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 $ _timestamp‚1.788442080147941e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51208516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37126103 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78929681 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510043036700160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37126525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_tr”zŸ;ain_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4520.193054998 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚611511281424384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611511281424384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚611513908025344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79627245 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591916302828544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚5399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51659117 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79627245 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89117201 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13734850927616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528304827718656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100645751 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528304902249472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514570023187456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚601916471839112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601497114199040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591916277400576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9997570725900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 _runtime‚4558.788335916 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100665785 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100645751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10016767434752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10016767434752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79627239 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574Ž£¢(€ˆP M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9997570725900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 _runtime‚4558.788335916 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100665785 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100645751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10016767434752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10016767434752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79627239 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 % _timestamp‚1.7884421187434251e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89117658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13734804531200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚611511281424384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611511281424384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚611513908025344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79627245 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591916302828544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚5399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51659117 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79627245 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89117201 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13734850927616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528304827718656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100645751 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528304902249472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514570023187456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚601916471839112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601497114199040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591916277400576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79641481 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚611623680072704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13737365130240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9999400068568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601605053904384 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51668313 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 EDeviceStatsMonitor.on_train_batch_end/inactive_split.alá.ækl.freed‚89133182 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚514662439375360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601605053904384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100663985 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚602024642356260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592022644002816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79641475 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100663985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10001998353444 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601605080295936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10015999554048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100684045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37464869 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528399804505600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79641475 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10018599776768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51668316 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚514662411240960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37465296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13737318747136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844+×Jª'ƒ’N _step‚442 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89133612 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚592022618574848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100684045 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528399729988096 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79641481 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚611623680072704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13737365130240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9999400068568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601605053904384 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51668313 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89133182 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚514662439375360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601605053904384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100663985 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚602024642356260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592022644002816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79641475 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100663985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10001998353444 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601605080295936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10015999554048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100684045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37464869 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528399804505600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79641475 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10018599776768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51668316 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚514662411240960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37465296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13737318747136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚602022018643416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884421194839714e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10018599776768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611621053458432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648ºû½éÇ2„- _wandbruntime‚4559  _runtime‚4559²Ÿ3kZ…Ô .loss/FRt4/detection_loss‚2.223532494961047 %loss/total_loss‚41.37874917521772 _runtime‚4559.530214941 1loss/FBt1/assignment_loss‚0.13853158955584943 /loss/FRt2/assignment_loss‚2.342762017250061 2loss/SRqqt3/assignment_loss‚1.6491888398709504 0loss/SRqqt4/detection_loss‚1.527314760589174 .loss/FBt2/detection_loss‚2.187755278178624 1loss/FBt4/assignment_loss‚0.22874405018228572 1loss/SRqqt1/detection_loss‚0.9671037048101425 1loss/SRqqt2/detection_loss‚1.8094376510381698 /loss/FBt4/detection_loss‚3.0014723390340805 trainer/global_step‚5399 /loss/FRt4/assignment_loss‚2.555449474091623 0loss/FBt2/assignment_loss‚0.4060666581083621 /loss/FRt3/detection_loss‚2.5169910160288156 epoch‚4 /loss/FRt1/assignment_loss‚1.420101240623829 0loss/FRt3/assignment_loss‚3.6380818582869865 2loss/SRqqt2/assignment_loss‚1.4389982578530907 1loss/SRqqt3/detection_loss‚3.0547777349534244 _step‚443 2loss/SRqqt1/assignment_loss‚0.8474539626778951 0loss/FBt3/assignment_loss‚0.1245942018693313 /loss/FRt1/detection_loss‚1.6342388583823693 /loss/FRt2/detection_loss‚2.6333264112472534 /loss/FBt1/detection_loss‚0.6020804143669766 % _timestamp‚1.7884421194856324e+09 2loss/SRqqt4/assignment_loss‚1.3377303686159263 .loss/FBt3/detection_loss‚3.093015992641449¼Š€¯_†Ù /loss/FRt1/assignment_loss‚1.420101240623829 0loss/FRt3/assignment_loss‚3.6380818582869865 2loss/SRqqt2/assignment_loss‚1.4389982578530907 1loss/SRqqt3/detection_loss‚3.0547777349534244 _step‚443 2loss/SRqqt1/assignment_loss‚0.8474539626778951 0loss/FBt3/assignment_loss‚0.1245942018693313 /loss/FRt1/detection_loss‚1.6342388583823693 /loss/FRt2/detection_loss‚2.6333264112472534 /loss/FBt1/detection_loss‚0.6020804143669766 % _timestamp‚1.7884421194856324e+09 2loss/SRqqt4/assignment_loss‚1.3377303686159263 .loss/FBt3/detection_loss‚3.093015992641449 .loss/FRt4/detection_loss‚2.223532494961047 %loss/total_loss‚41.37874917521772 _runtime‚4559.530214941 1loss/FBt1/assignment_loss‚0.13853158955584943 /loss/FRt2/assignment_loss‚2.342762017250061 2loss/SRqqt3/assignment_loss‚1.6491888398709504 0loss/SRqqt4/detection_loss‚1.527314760589174 .loss/FBt2/detection_loss‚2.187755278178624 1loss/FBt4/assignment_loss‚0.22874405018228572 1loss/SRqqt1/detection_loss‚0.9671037048101425 1loss/SRqqt2/detection_loss‚1.8094376510381698 /loss/FBt4/detection_loss‚3.0014723390340805 trainer/global_step‚5399 /loss/FRt4/assignment_loss‚2.555449474091623 0loss/FBt2/assignment_loss‚0.4060666581083621 /loss/FRt3/detection_loss‚2.5169910160288156 epoch‚4»¢æO“‡‚(:ˆ ‡ìåÔÆÂðÈ gpu.0.gpu‚84.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.695' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.339 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a OGˆ‚(:¼ ‡ìåÔ©ß˜Ë disk.rbd1.out‚ 2.146484375 disk.sda.in‚0.5625 disk.sda1.out‚0) disk./.usagePercent‚74.37113797274147( disk.nvme0c0n1.out‚42349.83349609375# disk.dm-0.out‚42349.83349609375 disk.sda.out‚ 173.61328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1754.04296875# proc.memory.rssMB‚ 2725.75390625& disk.nvme0n1.out‚42349.83349609375 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚375.51611328125) proc.memory.percent‚16.63668155670166 cpu‚25.307926078413352 network.sent‚ 1.2805492e+07 disk.sda3.out‚ 173.61328125 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 network.recv‚ 1.693435e+06 disk.dm-1.out‚ 173.61328125$ memory_percent‚89.08421993255615 proc.cpu.threads‚17% disk./.usageGB‚1108.1837005615234 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 1788.44140625Å’‰‚(:‡ –ìåÔ½ç€É gpu.0.gpu‚76.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚254.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ûü—@Š‚(:µ –ìåÔÕ‡‡Ë network.recv‚ 1.699372e+06 disk.sda3.in‚0.5625 proc.cpu.threads‚17 network.sent‚ 1.2913514e+07! disk.rbd2.out‚381.77392578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 174.015625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚42447.55908203125 disk.dm-1.in‚0.5625& disk.nvme0n1.out‚42447.55908203125 disk.dm-1.out‚ 174.015625$ memory_percent‚89.08495903015137 cpu‚25.307086935249753% disk./.usageGB‚1108.1836585998535 disk.rbd1.in‚0# proc.memory.rssMB‚ 2725.75390625# disk.dm-0.out‚42447.55908203125 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875( proc.memory.availableMB‚ 1788.3203125) disk./.usagePercent‚74.37113515665858 disk.rbd1.out‚ 2.146484375 disk.sda.out‚ 174.015625 disk.rbd0.out‚0) proc.memory.percent‚16.63668155670166 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1754.04296875 disk.sda1.in‚0 disk.rbd3.out‚0~ÇqBŸ‹‚(:” ¥ìåÔíàòÈ gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚255.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.093599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OËS]DŒ‚(:¹ ¥ìåÔŽôÒÊ disk.sda3.in‚0.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚42447.82080078125# disk.dm-0.out‚42447.82080078125 disk.rbd1.in‚0 network.sent‚ 1.2919546e+07# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.rbd1.out‚ 2.146484375 disk.rbd0.in‚0 disk.rbd2.out‚389.3564453125) disk./.usagePercent‚74.37113515665858 disk.rbd0.out‚0 disk.dm-1.out‚ 174.51953125 disk.rbd2.in‚0 disk.sda.in‚0.5625( proc.memory.availableMB‚ 1788.3359375 disk.sda3.out‚ 174.51953125 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.sda.out‚ 174.51953125) proc.memory.percent‚16.63668155670166 cpu‚25.30620220490896 proc.cpu.threads‚17 network.recv‚ 1.703548e+06 disk.sda2.out‚0& disk.nvme0n1.out‚42447.82080078125 disk.drbd1777.in‚0$ memory_percent‚89.08486366271973# proc.memory.rssMB‚ 2725.75390625% disk./.usageGB‚1108.1836585998535 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0綈`2- _wandbruntime‚4598  _runtime‚4598¶ŠºlÉŽÃ _runtime‚4598.106990506 _step‚444 ' lr-AdamW/pg1‚0.00025957882989052306 ' lr-AdamW/pg2‚0.00025957882989052306 trainer/global_step‚5449 % _timestamp‚1.7884421580623446e+09}ÌhfÎÈ ' lr-AdamW/pg1‚0.00025957882989052306 ' lr-AdamW/pg2‚0.00025957882989052306 trainer/global_step‚5449 % _timestamp‚1.7884421580623446e+09 _runtime‚4598.106990506 _step‚444¼ž{†2- _wandbruntime‚4598  _runtime‚4598‡u„R˜$‘ƒP TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37796810 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533050361550848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚445 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚617002524141568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607325011277320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10091649735176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101580837 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101560803 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606894099466240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚617002524141568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101560803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10089051450508 trainer/global_step‚5449 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10108398283776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101580837 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚606894125857792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52118920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607322387564684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519189425862656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597233336114176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519189453997056 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80339039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616999897540608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606894099466240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37797264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 $ _timestamp‚1.788442158063013e+09 _runtime‚4598.107910981 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80339045 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚606894125857792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89916184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52118917 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10108398283776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89915727 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13860935688192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533050436081664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616999897540608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80339045 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13860982084608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonito2†Èqr.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597233361542144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616999897540608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606894099466240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37797264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 $ _timestamp‚1.788442158063013e+09 _runtime‚4598.107910981 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80339045 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚606894125857792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89916184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52118917 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10108398283776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89915727 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13860935688192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533050436081664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616999897540608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80339045 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13860982084608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597233361542144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606894099466240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚617002524141568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101560803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10089051450508 trainer/global_step‚5449 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10108398283776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101580837 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚606894125857792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52118920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607322387564684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519189425862656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597233336114176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574½léâÌ2“- _wandbruntime‚4598  _runtime‚4598cÝ‚'”ŽN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80353281 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101599097 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607430558081624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607002065563136 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101599097 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚607002039171584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10093479078052 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101579037 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚607002039171584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181932312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 trainer/global_step‚5449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37803981 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89932097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80353275 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80353275 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52128113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597339677288448 _step‚446 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4598.843382922 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101579037 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10110230625792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52128116 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10090880793176 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607433181794468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884421587979817e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533145334659072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89931644 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚617109669574656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519281813916160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10107630403072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13863492608512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074EßÀž'•“N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80353281 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101599097 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607430558081624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607002065563136 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101599097 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚607002039171584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10093479078052 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101579037 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚607002039171584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181932312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 trainer/global_step‚5449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37803981 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89932097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80353275 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80353275 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52128113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597339677288448 _step‚446 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4598.843382922 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101579037 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10110230625792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52128116 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10090880793176 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607433181794468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 % _timestamp‚1.7884421587979817e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533145334659072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89931644 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚617109669574656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519281813916160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10107630403072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13863492608512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378¾ðpI2–- _wandbruntime‚4598  _runtime‚4598”…ìk[—Õ 2loss/SRqqt1/assignment_loss‚0.8648871617078839 2loss/SRqqt4/assignment_loss‚1.3349078033330306 1loss/FBt1/assignment_loss‚0.14719001839781218 1loss/SRqqt1/detection_loss‚1.0214776815264486 0loss/FRt2/assignment_loss‚3.6118158088173975 2loss/SRqqt2/assignment_loss‚1.6576748684462574 /loss/FRt1/detection_loss‚1.6062292867275172 /loss/FBt4/detection_loss‚2.6650323271751404 epoch‚4 trainer/global_step‚5449 _step‚447 /loss/FRt4/assignment_loss‚3.143085964024067 2loss/SRqqt3/assignment_loss‚2.0094436113323484 1loss/FBt4/assignment_loss‚0.07581176671374124 /loss/FRt2/detection_loss‚2.3823088130285575 0loss/SRqqt4/detection_loss‚1.655879741569735 _runtime‚4598.844515026 0loss/FRt1/assignment_loss‚0.9357669090115747 .loss/FRt3/detection_loss‚2.507051333105652 /loss/FRt4/detection_loss‚2.6898918822407722 1loss/FBt2/assignment_loss‚0.23344365840312092 0loss/SRqqt3/detection_loss‚2.282585448878152 /loss/FBt1/detection_loss‚0.5528965806019535 0loss/FBt3/assignment_loss‚0.3055161119139354 /loss/FBt2/detection_loss‚2.6996180057525634 /loss/FBt3/detection_loss‚2.6987463321004594 -loss/FRt3/assignment_loss‚3.5383220595472 %loss/total_loss‚42.66427041898253 % _timestamp‚1.7884421587999208e+09 1loss/SRqqt2/detection_loss‚2.0446872446272106sF]˜Ú 1loss/SRqqt2/detection_loss‚2.0446872446272106 2loss/SRqqt1/assignment_loss‚0..‘AW8648871617078839 2loss/SRqqt4/assignment_loss‚1.3349078033330306 1loss/FBt1/assignment_loss‚0.14719001839781218 1loss/SRqqt1/detection_loss‚1.0214776815264486 0loss/FRt2/assignment_loss‚3.6118158088173975 2loss/SRqqt2/assignment_loss‚1.6576748684462574 /loss/FRt1/detection_loss‚1.6062292867275172 /loss/FBt4/detection_loss‚2.6650323271751404 epoch‚4 trainer/global_step‚5449 _step‚447 /loss/FRt4/assignment_loss‚3.143085964024067 2loss/SRqqt3/assignment_loss‚2.0094436113323484 1loss/FBt4/assignment_loss‚0.07581176671374124 /loss/FRt2/detection_loss‚2.3823088130285575 0loss/SRqqt4/detection_loss‚1.655879741569735 _runtime‚4598.844515026 0loss/FRt1/assignment_loss‚0.9357669090115747 .loss/FRt3/detection_loss‚2.507051333105652 /loss/FRt4/detection_loss‚2.6898918822407722 1loss/FBt2/assignment_loss‚0.23344365840312092 0loss/SRqqt3/detection_loss‚2.282585448878152 /loss/FBt1/detection_loss‚0.5528965806019535 0loss/FBt3/assignment_loss‚0.3055161119139354 /loss/FBt2/detection_loss‚2.6996180057525634 /loss/FBt3/detection_loss‚2.6987463321004594 -loss/FRt3/assignment_loss‚3.5383220595472 %loss/total_loss‚42.66427041898253 % _timestamp‚1.7884421587999208e+09¿¾˜W’™‚(:‡ ´ìåÔçÈÉ gpu.0.gpu‚68.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚249.78' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ðÿBš‚(:· ´ìåÔ΢©Ë network.sent‚ 1.2925349e+07 network.recv‚ 1.707848e+06 disk.rbd0.out‚0 disk.rbd2.out‚396.6884765625 disk.sda1.in‚0! disk.nvme0n1.in‚ 1754.04296875% disk.nvme0n1.out‚42545.5537109375" disk.dm-0.out‚42545.5537109375 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.146484375 disk.rbd3.out‚0) disk./.usagePercent‚74.37115154114078 disk.dm-1.out‚ 175.01953125$ memory_percent‚89.08612728118896 disk.sda3.out‚ 175.01953125' disk.nvme0c0n1.out‚42545.5537109375 disk.drbd1777.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.rbd1.in‚0 disk.sda.out‚ 175.01953125 disk.sda2.in‚0) proc.memory.availableMB‚ 1788.12890625) proc.memory.percent‚16.63668155670166 cpu‚25.30536481081339% disk./.usageGB‚1108.1839027404785 disk.rbd0.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2725.75390625 disk.sda3.in‚0.5625 disk.rbd2.in‚0¥#ÿž›‚(:“ ÃìåÔ ¬§É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.829' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.36580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”:Î=œ‚(:² ÃìåÔ™ýÍË disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0) proc.memory.percent‚16.63668155670166 disk.rbd0.out‚0# disk.dm-0.out‚42545.83740234375 disk.dm-1.out‚ 175.484375 disk.rbd1.in‚0 disk.sda.out‚ 175.484375 disk.sda2.out‚0 network.recv‚ 1.71477e+06 disk.rbd0.in‚0 cpu‚25.30448345275602! disk.nvme0n1.in‚ 1754.04296875 disk.rbd2.in‚0 disk.rbd3.out‚0$ memory_percent‚89.08698558807373) proc.memory.availableMB‚ 1787.98828125 network.sent‚ 1.3034527e+07 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚42545.83740234375 disk.dm-0.in‚ 1754.04296875) disk./.usagePercent‚74.37115947737433# disk.nvme0c0n1.in‚ 1754.04296875 proc.cpu.threads‚17% disk./.usageGB‚1108.1840209960938& disk.nvme0n1.out‚42545.83740234375 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda.in‚0.5625 disk.sda2.in‚0# proc.memory.rssMB‚ 2725.75390625 disk.sda3.out‚ 175.484375 disk.drbd1777.out‚0 disk.rbd2.out‚ 404.533203125‡šœ»‘‚(:† ÒìåÔ¦£ÞÈ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚42.0 gpu.0.powerWatts‚91.498' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽŒ\WEž‚(:º ÒìåÔè³ëË disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 176.11328125# disk.dm-0.out‚42546.57275390625 disk.dm-1.in‚0.5625 disk.sda1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚42546.57275390625 disk.rbd2.out‚ 411.865234375$ memory_percent‚89.08698558807373 disk.sda3.in‚0.5625 disk.drbd1777.out‚0 disk.sda2.in‚0 cpu‚25.303662232406904 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 176.11328125! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.146484375& disk.nvme0n1.out‚42546.57275390625) proc.memory.availableMB‚ 1787.98828125) proc.memory.percent‚16.63668155670166 network.sent‚ 1.3039839e+07 network.recv‚ 1.718601e+06% disk./.usageGB‚1108.1838760375977 disk.sda.in‚0.5625 disk.sda.out‚ 176.11328125 disk.sda2.out‚0# proc.memory.rssMB‚ 2725.75390625) disk./.usagePercent‚74.37114974908803 disk.rbd1.in‚0 disk.rbd2.in‚0Õ¬1Ñ2Ÿ- _wandbruntime‚4637  _runtime‚4637ÂD(Ç Á trainer/global_step‚5499 % _timestamp‚1.7884421972229474e+09 _runtime‚4637.267689928 _step‚448 & lr-AdamW/pg1‚0.0002569413050648264 & lr-AdamW/pg2‚0.0002569413050648264Œ/[Ì¡Æ % _timestamp‚1.7884421972229474e+09 _runtime‚4637.267689928 _step‚448 & lr-AdamW/pg1‚0.0002569413050648264 & lr-AdamW/pg2‚0.0002569413050648264 trainer/global_step‚5499À‘:ÌÝ2¢- _wandbruntime‚4637  _runtime‚4637®•Îß (£ƒP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102495889 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10197428923392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚537795755065856 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13986898393600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚622488513656832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10200029132800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81050839 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612733550715528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5499 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10197428923392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81050845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52578717 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102475855 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 _runtime‚4637.268850105 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚622491140257792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38135207 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523808856672256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52578720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13986851997184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602550394827776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90713924 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚612291111124992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612730927002892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38135668 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚622488513656832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚449 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10200029132800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102495889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537795680535040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚612291111124992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚622488513656832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚449 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10200029132800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102495889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537795680535040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚612291111124992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚622488513656832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10200029132800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81050839 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612733550715528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5499 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10197428923392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81050845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52578717 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102475855 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 _runtime‚4637.268850105 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚622491140257792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38135207 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523808856672256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52578720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13986851997184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602550394827776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90713924 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚612291111124992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612730927002892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÁÚ–†+2¥- _wandbruntime‚4638  _runtime‚4638w‰9»Ï¦N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚612399050830336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102494089 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 _runtime‚4638.003139005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102494089 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523901216591360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10201861474816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523901244725760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81065075 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102514149 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602656761430016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10182361517784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 _step‚450 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52587913 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚612399050830336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38142314 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612841721232676 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90730230 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102514149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38141870 WGDevice(WÝDStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚612399024438784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10184959802660 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 $ _timestamp‚1.788442197957996e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81065081 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚622598285690880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚537890596035584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81065075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602656736002048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚622600912305152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚612399024438784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90729783 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612839097519832 trainer/global_step‚5499 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52587916 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81065081 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183579230 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90729783 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612839097519832 trainer/global_step‚5499 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52587916 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81065081 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183579230 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523901216591360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10201861474816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523901244725760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81065075 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102514149 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602656761430016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10182361517784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 _step‚450 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183559164 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52587913 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚612399050830336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38142314 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612841721232676 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90730230 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102514149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38141870 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚612399024438784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10184959802660 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 $ _timestamp‚1.788442197957996e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81065081 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚622598285690880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚537890596035584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81065075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602656736002048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104ÂXbÊU2¨- _wandbruntime‚4638  _runtime‚4638É`wQ]©× 2loss/SRqqt4/assignment_loss‚1.4992076411367732 2loss/FBt4/assignment_loss‚0.005147298565134406 % _timestamp‚1.7884421979601483e+09 _step‚451 trainer/global_step‚5499 0loss/FRt3/assignment_loss‚3.6458124352826013 1loss/SRqqt3/assignment_loss‚2.398757799556761 1loss/FBt3/assignment_loss‚0.25869907122493413 /loss/FBt1/detection_loss‚0.6815664226797035 epoch‚4 0loss/FRt4/assignment_loss‚2.7997819839739333 .loss/FRt3/detection_loss‚2.621269653458987 0loss/SRqqt1/detection_loss‚1.200713792017528 _runtime‚4638.004760541 0loss/FRt2/assignment_loss‚2.9016332719840254 1loss/FBt1/assignment_loss‚0.16341245958465467 1loss/SRqqt3/detection_loss‚2.1113186568924873 /loss/FBt2/detection_loss‚2.8278832510113716 2loss/SRqqt1/assignment_loss‚1.0880074297329263 1loss/FBt2/assignment_loss‚0.07020549001754262 /loss/FRt1/detection_loss‚1.9928742004153521 1loss/SRqqt2/detection_loss‚1.7485081823221569 %loss/total_loss‚44.59920188446512 0loss/FRt1/assignment_loss‚1.4975141818108766 /loss/FRt2/detection_loss‚2.2538524702483533 .loss/FBt3/detection_loss‚4.201650242010753 .loss/FRt4/detection_loss‚2.293519860389186 1loss/SRqqt4/detection_loss‚1.7606360065810225 .loss/FBt4/detection_loss‚2.781764038971492 1loss/SRqqt2/assignment_loss‚1.795466044596557eÿÁ[bªÜ 1loss/FBt2/assignment_loss‚0.07020549001754262 /loss/FRt1/detection_loss‚1.9928742004153521 1loss/SRqqt2/detection_loss‚1.7485081823221569 %loss/total_loss‚44.59920188446512 0loss/FRt1/assignment_loss‚1.4975141818108766 /loss/FRt2/detection_loss‚2.2538524702483533 .loss/FBt3/detection_loss‚4.201650242010753 .loss/FRt4/detection_loss‚2.293519860389186 1loss/SRqqt4/detection_loss‚1.7606360065810225 .loss/FBt4/detection_loss‚2.781764038971492 1loss/SRqqt2/assignment_loss‚1.795466044596557 2loss/SRqqt4/assignment_loss‚1.4992076411367732 2loss/FBt4/assignment_loss‚0.005147298565134406 % _timestamp‚1.7884421979601483e+09 _step‚451 trainer/global_step‚5499 0loss/FRt3/assignment_loss‚3.6458124352826013 1loss/SRqqt3/assignment_loss‚2.398757799556761 1loss/FBt3/assignment_loss‚0.25869907122493413 /loss/FBt1/detection_loss‚0.6815664226797035 epoch‚4 0loss/FRt4/assignment_loss‚2.7997819839739333 .loss/FRt3/detection_loss‚2.621269653458987 0loss/SRqqt1/detection_loss‚1.200713792017528 _runtime‚4638.004760541 0loss/FRt2/assignment_loss‚2.9016332719840254 1loss/FBt1/assignment_loss‚0.16341245958465467 1loss/SRqqt3/detection_loss‚2.1113186568924873 /loss/FBt2/detection_loss‚2.8278832510113716 2loss/SRqqt1/assignment_loss‚1.0880074297329263à º;Ï”«‚(:‰ áìåÔ©ŠÉ gpu.0.gpu‚79.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚104.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%U'8¬‚(:­ áìåÔÙòâË disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1754.04296875 network.sent‚ 1.3045844e+07 disk.dm-1.out‚ 176.66796875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 176.66796875 proc.memory.availableMB‚1788 proc.cpu.threads‚17) disk./.usagePercent‚74.37115128513324 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2725.75390625% disk./.usageGB‚1108.1838989257812 disk.rbd2.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.04296875# disk.dm-0.out‚42644.07666015625 disk.rbd2.out‚ 419.427734375 disk.rbd3.in‚0& disk.nvme0n1.out‚42644.07666015625 disk.sda3.in‚0.5625 disk.rbd0.out‚0 network.recv‚ 1.722678e+06 disk.rbd1.out‚ 2.146484375 disk.sda.in‚0.5625 disk.sda.out‚ 176.66796875 disk.sda2.in‚0( disk.nvme0c0n1.out‚42644.07666015625) proc.memory.percent‚16.63668155670166 disk.rbd1.in‚0 cpu‚25.302795902670383 disk.dm-1.in‚0.5625! disk.nvme0n1.in‚ 1754.04296875 memory_percent‚ 89.0869140625…±ÝÓ­‚(:ˆ ðìåÔͬ—É gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.935' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.987 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m.ÍB®‚(:· ðìåÔ í®Ì proc.cpu.threads‚17 network.recv‚ 1.729132e+06! disk.nvme0n1.in‚ 1754.04296875 disk.sda3.out‚ 177.16015625 disk.rbd0.out‚0 disk.sda.in‚0.5625# proc.memory.rssMB‚ 2725.75390625$ disk./.usageGB‚1108.183879852295 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚89.08681869506836 cpu‚25.301981011590907 network.sent‚ 1.3154367e+07 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚42644.62255859375 disk.dm-1.in‚0.5625 disk.rbd1.in‚0' proc.memory.availableMB‚ 1788.015625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.sda.out‚ 177.16015625) proc.memory.percent‚16.63668155670166# disk.dm-0.out‚42644.62255859375 disk.rbd1.out‚ 2.146484375 disk.rbd2.out‚ 426.759765625) disk./.usagePercent‚74.37115000509557 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚42644.62255859375 disk.dm-1.out‚ 177.16015625|uÇ2¯- _wandbruntime‚4676  _runtime‚4676Fï2ÍÈ°Â _runtime‚4676.150723431 _step‚452 ' lr-AdamW/pg1‚0.00025429061273445736 ' lr-AdamW/pg2‚0.00025429061273445736 trainer/global_step‚5549 $ _timestamp‚1.788442236106031e+09A–n-걂 ' lr-AdamW/pg2‚0.00025429061273445736 trainer/global_step‚5549 $ _timestamp‚1.788442236106031e+09 _runtime‚4676.150723431 _step‚452 ' lr-AdamW/pg1‚0.00025429061273445736ă,†!2²- _wandbruntime‚4676  _runtime‚4676i·È (³„P PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81762645 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528428259347456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 trainer/global_step‚5549 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627977129773056 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103410941 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10289059772416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10274611184392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14112852367872 _step‚453 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617688070000640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81762639 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542541037184512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14112805971456 _runtime‚4676.151744365 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91509799 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91510249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53038520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10291659981824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627979756374016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚617688070000640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542541111715328 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚617688096392192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617688096392192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103410941 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103390907 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10289059772416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚618139466441100 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607867478969344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38471282eÔf*¦´‰P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚617688096392192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617688096392192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103410941 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103390907 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10289059772416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚618139466441100 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607867478969344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38471282 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81762645 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528428259347456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WÿÓ”ïi GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 trainer/global_step‚5549 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627977129773056 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103410941 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10289059772416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10274611184392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14112852367872 _step‚453 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617688070000640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81762639 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542541037184512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14112805971456 _runtime‚4676.151744365 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91509799 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91510249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53038520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10291659981824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627979756374016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚617688070000640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542541111715328 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574Åxíì;2µ- _wandbruntime‚4676  _runtime‚4676ÕfZ'¶ŽN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38478309 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚618247636958040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607973794715648 _runtime‚4676.889225279 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81776881 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528520647400960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14115369833472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚5549 % _timestamp‚1.7884422368441339e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚454 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚618250260670884 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10293492323840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81776881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚628089528421376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81776875 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81776875 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617796009705984 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10293492323840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185186016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14115323450368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617796036097536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91525594 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607973820143616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103409141 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10290892101120 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53047716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53047713 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103429201 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3#a'·“N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185186016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14115323450368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617796036097536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91525594 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607973820143616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103409141 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10290892101120 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53047716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53047713 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103429201 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38478309 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚618247636958040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607973794715648 _runtime‚4676.889225279 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81776881 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528520647400960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14115369833472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚5549 % _timestamp‚1.7884422368441339e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚454 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚618250260670884 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10293492323840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81776881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚628089528421376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81776875 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81776875 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617796009705984 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10293492323840Æú E2¸- _wandbruntime‚4676  _runtime‚4676*í9ëY¹Ó % _timestamp‚1.7884422368458056e+09 /loss/FRt2/assignment_loss‚2.921446909775605 /loss/FBt3/detection_loss‚3.1447950839996337 1loss/SRqqt4/detection_loss‚1.8557954142659396 2loss/SRqqt4/assignment_loss‚1.3002689000282248 .loss/FRt3/detection_loss‚2.404843833609676 /loss/FRt4/detection_loss‚2.3576350578894982 epoch‚4 2loss/SRqqt3/assignment_loss‚1.8067024284072462 1loss/SRqqt1/detection_loss‚0.8593271878966386 _runtime‚4676.890706751 /loss/FRt1/assignment_loss‚2.085693768426484 0loss/SRqqt3/detection_loss‚2.591879783138152 /loss/FBt4/detection_loss‚2.0916133175293603 $loss/total_loss‚43.2755479302972 2loss/SRqqt2/assignment_loss‚1.3627999424934387 1loss/FBt3/assignment_loss‚0.03688624361529946 /loss/FRt1/detection_loss‚1.7395055911996786 .loss/FRt2/detection_loss‚2.520022978653779 1loss/SRqqt2/detection_loss‚1.7078023807090872 /loss/FBt1/detection_loss‚0.7789869723764921 /loss/FBt2/detection_loss‚2.8460024495919547 /loss/FRt4/assignment_loss‚2.629104610819083 1loss/SRqqt1/assignment_loss‚0.989474029806645 1loss/FBt1/assignment_loss‚0.09939173158618102 trainer/global_step‚5549 _step‚455 /loss/FRt3/assignment_loss‚4.060162215762668 1loss/FBt2/assignment_loss‚0.01770488896484797 0loss/FBt4/assignment_loss‚1.0677022097515874Mjg±^ºØ 2loss/SRqqt4/assignment_loss‚1.3002689000282248 .loss/FRt3/detection_loss‚2.404843833609676 /loss/FRt4/detection_loss‚2.3576350578894982 epoch‚4 2loss/SRqqt3/assignment_loss‚1.8067024284072462 1loss/SRqqt1/detection_loss‚0.8593271878966386 _runtime‚4676.890706751 /loss/FRt1/assignment_loss‚2.085693768426484 0loss/SRqqt3/detection_loss‚2.591879783138152 /loss/FBt4/detection_loss‚2.0916133175293603 $loss/total_loss‚43.2755479302972 2loss/SRqqt2/assignment_loss‚1.3627999424934387 1loss/FBt3/assignment_loss‚0.03688624361529946 /loss/FRt1/detection_loss‚1.7395055911996786 .loss/FRt2/detection_loss‚2.520022978653779 1loss/SRqqt2/detection_loss‚1.7078023807090872 /loss/FBt1/detection_loss‚0.7789869723764921 /loss/FBt2/detection_loss‚2.8460024495919547 /loss/FRt4/assignment_loss‚2.629104610819083 1loss/SRqqt1/assignment_loss‚0.989474029806645 1loss/FBt1/assignment_loss‚0.09939173158618102 trainer/global_step‚5549 _step‚455 /loss/FRt3/assignment_loss‚4.060162215762668 1loss/FBt2/assignment_loss‚0.01770488896484797 0loss/FBt4/assignment_loss‚1.0677022097515874 % _timestamp‚1.7884422368458056e+09 /loss/FRt2/assignment_loss‚2.921446909775605 /loss/FBt3/detection_loss‚3.1447950839996337 1loss/SRqqt4/detection_loss‚1.8557954142659396Çõó+*ž»‚(:“ ÿìåÔµû°É gpu.0.gpu‚69.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.624' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.12480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õvUC¼‚(:¸ ÿìåÔÛ¥Ì( disk./.usagePercent‚74.3711538452086 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚42742.09130859375) proc.memory.availableMB‚ 1787.74609375$ disk./.usageGB‚1108.183937072754 disk.sda1.in‚0 disk.sda3.in‚0.5625 cpu‚25.30117967589162 disk.rbd2.in‚0 disk.sda.in‚0.5625# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-1.out‚ 177.73046875 proc.cpu.threads‚17 network.recv‚ 1.733868e+06 disk.rbd1.out‚ 2.146484375 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.out‚434.4052734375! disk.nvme0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.sda1.out‚0 network.sent‚ 1.3160595e+07 disk.sda.out‚ 177.73046875 disk.sda3.out‚ 177.73046875 disk.rbd0.in‚0# disk.dm-0.out‚42742.09130859375$ memory_percent‚89.08846378326416 disk.drbd1777.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚42742.09130859375%i„u(½‚(:“ ŽíåÔ¶ë²É gpu.0.gpu‚~Wòlv48.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚42.0 gpu.0.powerWatts‚91.604' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.320800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôfÂ,B¾‚(:· ŽíåÔÞ½ëÊ disk.rbd0.in‚0" disk.dm-0.out‚42742.9208984375 disk.rbd1.out‚ 2.146484375 disk.rbd2.in‚0 disk.rbd2.out‚441.7373046875 disk.sda.out‚ 178.08203125 disk.rbd3.in‚0# memory_percent‚89.0864610671997 network.sent‚ 1.3268544e+07) disk./.usagePercent‚74.37116178144215% disk./.usageGB‚1108.1840553283691 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.63668155670166 disk.dm-1.in‚0.5625 proc.cpu.threads‚17# proc.memory.rssMB‚ 2725.75390625 disk.sda3.out‚ 178.08203125# disk.nvme0c0n1.in‚ 1754.04296875 network.recv‚ 1.739165e+06 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.drbd1777.in‚0% disk.nvme0n1.out‚42742.9208984375' disk.nvme0c0n1.out‚42742.9208984375 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 178.08203125 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd0.out‚0) proc.memory.availableMB‚ 1788.07421875 cpu‚25.300344098203208¼7vÒ”¿‚(:‰ íåÔ±Ú®É gpu.0.gpu‚72.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚248.998' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™^=À‚(:² íåÔºâ®Ë disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1788.08203125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.out‚ 178.703125 network.sent‚ 1.3273792e+07 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd2.out‚442.8779296875 disk.sda.in‚0.5625% disk.nvme0n1.out‚42743.2060546875 network.recv‚ 1.743871e+06" disk.dm-0.out‚42743.2060546875 disk.dm-1.in‚0.5625 disk.sda.out‚ 178.703125 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0' disk.nvme0c0n1.out‚42743.2060546875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.294921875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 178.703125$ memory_percent‚89.08641338348389) disk./.usagePercent‚74.35457070117064 disk.dm-0.in‚ 1754.04296875# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 cpu‚25.299545545327206% disk./.usageGB‚1107.9368362426758# disk.nvme0c0n1.in‚ 1754.04296875Áòï–2Á- _wandbruntime‚4715  _runtime‚4715!’çÀÇÂÁ & lr-AdamW/pg1‚0.0002516273052997246 & lr-AdamW/pg2‚0.0002516273052997246 trainer/global_step‚5599 % _timestamp‚1.7884422752440646e+09 _runtime‚4715.288719289 _step‚456˜É+tÌÃÆ & lr-AdamW/pg1‚0.0002516273052997246 & lr-AdamW/pg2‚0.0002516273052997246 trainer/global_step‚5599 % _timestamp‚1.7884422752440646e+09 _runtime‚4715.288719289 _step‚456È]kDI2Ä- _wandbruntime‚4715  _runtime‚4715e¶Óï (Å„P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547286726908928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613184512254976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633468372490240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82474439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 trainer/global_step‚5599 % _timestamp‚1.7884422752448297e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104305959 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92305347 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14239137319936 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚457 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633468372490240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38807493 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623085081659392 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10383290830848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82474445 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104305959 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104325993 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633465745889280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38807030 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613184537682944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10366091909000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10363493624332 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186780398 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623085055267840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10380690621440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚533047662022656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 _runtime‚4715.289756213 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104325993 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚623550629591944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚623085081659392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10383290830848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82474445 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633465745889280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53498320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53498317 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚623548005879308 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚533047633888256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14239093020672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10380690621440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0cD­(ƉP VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104305959 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92305347 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14239137319936 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _step‚457 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633468372490240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38807493 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623085081659392 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10383290830848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82474445 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104305959 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104325993 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633465745889280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38807030 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613184537682944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10366091909000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10363493624332 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186780398 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623085055267840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10380690621440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚533047662022656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 _runtime‚4715.289756213 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104325993 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚623550629591944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚623085081659392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10383290830848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82474445 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633465745889280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53498320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53498317 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚623548005879308 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚533047633888256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14239093020672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10380690621440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186800438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547286726908928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613184512254976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633468372490240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82474439 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 trainer/global_step‚5599 % _timestamp‚1.7884422752448297e+09ÉY}52Ç- _wandbruntime‚4716  _runtime‚4716²I³3á!ÈŽN _step‚458 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623193021364736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633578144537600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10365322967000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547381723241984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10367921251876 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10382522950144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚623656176396248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623192994973184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38814225 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613290878857216 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92321298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14241673165824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚623658800109092 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104324193 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104324193 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82488675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53507513 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚533140021941760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚613290853429248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633578144537600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38813785 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82488675 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82488681 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚5599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10385123172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633575517923328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚623192994973184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10385123172864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allçd3ocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53507516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533140050076160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 _runtime‚4716.029873448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82488681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚547381648724480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92321741 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14241626782720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074ª.® 'É“N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚633578144537600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10365322967000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547381723241984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10367921251876 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10382522950144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚623656176396248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623192994973184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38814225 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613290878857216 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92321298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14241673165824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚623658800109092 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104324193 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104324193 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82488675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53507513 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚533140021941760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚613290853429248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633578144537600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38813785 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82488675 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82488681 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 trainer/global_step‚5599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10385123172864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633575517923328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚623192994973184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10385123172864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53507516 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533140050076160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 _runtime‚4716.029873448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82488681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚547381648724480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92321741 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14241626782720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚458 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623193021364736ÊÛô1K2Ê- _wandbruntime‚4716  _runtime‚4716ò úVËÐ 1loss/SRqqt2/detection_loss‚1.8433057232981636 .loss/FBt1/detection_loss‚0.587786790980574 % _timestamp‚1.7884422759863658e+09 trainer/global_step‚5599 1loss/SRqqt3/assignment_loss‚1.881550463683465 1loss/FBt3/assignment_loss‚0.11479898833567859 1loss/FBt4/assignment_loss‚0.26070237561361864 ,loss/FBt3/detection_loss‚3.0246203975244 1loss/FBt1/assignment_loss‚0.07644300640355368 %loss/total_loss‚43.18183763388403 _step‚459 .loss/FRt1/detection_loss‚1.446883397120418 0loss/FRt1/assignment_loss‚1.2180736245531025 .loss/FBt4/detection_loss‚2.344658136367798 epoch‚4 1loss/SRqqt4/detection_loss‚1.8475213679028493 /loss/FRt3/assignment_loss‚3.887303722315821 0loss/FRt4/assignment_loss‚3.3304921131508025 0loss/SRqqt3/detection_loss‚2.309807926416397 /loss/FBt2/detection_loss‚3.2466551065444946 /loss/FRt2/assignment_loss‚3.010682017245191 1loss/FBt2/assignment_loss‚0.01210970925167203 .loss/FRt3/detection_loss‚2.625682534126634 2loss/SRqqt1/assignment_loss‚1.0428103833385791 2loss/SRqqt2/assignment_loss‚1.5906471313819999 _runtime‚4716.030925081 1loss/SRqqt4/assignment_loss‚1.556647708529935 /loss/FRt2/detection_loss‚2.4886679383034402 .loss/FRt4/detection_loss‚2.404192097046796 1loss/SRqqt1/detection_loss‚1.0297949744486494m¬»[ÌÕ 1loss/SRqqt4/detection_loss‚1.8475213679028493 /loss/FRt3/assignment_loss‚3.887303722315821 0loss/FRt4/assignment_loss‚3.3304921131508025 0loss/SRqqt3/detection_loss‚2.309807926416397 /loss/FBt2/detection_loss‚3.2466551065444946 /loss/FRt2/assignment_loss‚3.010682017245191 1loss/FBt2/assignment_loss‚0.01210970925167203 .loss/FRt3/detection_loss‚2.625682534126634 2loss/SRqqt1/assignment_loss‚1.0428103833385791 2loss/SRqqt2/assignment_loss‚1.5906471313819999 _runtime‚4716.030925081 1loss/SRqqt4/assignment_loss‚1.556647708529935 /loss/FRt2/detection_loss‚2.4886679383034402 .loss/FRt4/detection_loss‚2.404192097046796 1loss/SRqqt1/detection_loss‚1.0297949744486494 1loss/SRqqt2/detection_loss‚1.8433057232981636 .loss/FBt1/detection_loss‚0.587786790980574 % _timestamp‚1.7884422759863658e+09 trainer/global_step‚5599 1loss/SRqqt3/assignment_loss‚1.881550463683465 1loss/FBt3/assignment_loss‚0.11479898833567859 1loss/FBt4/assignment_loss‚0.26070237561361864 ,loss/FBt3/detection_loss‚3.0246203975244 1loss/FBt1/assignment_loss‚0.07644300640355368 %loss/total_loss‚43.18183763388403 _step‚459 .loss/FRt1/detection_loss‚1.446883397120418 0loss/FRt1/assignment_loss‚1.2180736245531025 .loss/FBt4/detection_loss‚2.344658136367798 epoch‚4Ë Þ¾9ŸÍ‚(:” ¬íåÔù–©É gpu.0.gpu‚56.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.909' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.781800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅŽ³a8΂(:­ ¬íåÔÒ¼Ì) disk./.usagePercent‚74.30162629502948 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚43004.423828125 disk.rbd2.out‚443.3896484375 disk.sda1.in‚0! disk.nvme0n1.in‚ 1754.04296875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.75390625 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.04296875! disk.dm-0.out‚43004.423828125) proc.memory.percent‚16.63668155670166 cpu‚25.298745587763477 network.sent‚ 1.3280001e+07 network.recv‚ 1.74883e+06 disk.rbd1.out‚ 2.294921875 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚43004.423828125 disk.dm-1.out‚ 179.015625$ memory_percent‚89.08648490905762% disk./.usageGB‚1107.1479263305664 disk.drbd1777.out‚0 disk.sda.out‚ 179.015625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.5625( proc.memory.availableMB‚ 1788.0703125 disk.sda1.out‚0 disk.sda3.out‚ 179.015625 disk.dm-0.in‚ 1754.04296875 proc.cpu.threads‚17ŸÓ» žÏ‚(:“ »íåÔÚ’·É gpu.0.gpu‚67.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚270.136' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.02720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j»þ6Ђ(:« »íåÔÎáúÊ! disk.nvme0n1.in‚ 1754.04296875$ disk.nvme0n1.out‚43004.716796875 disk.sda3.in‚0.5625 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 179.78125 cpu‚25.29792709845849 network.recv‚ 1.755287e+06 disk.sda.in‚0.5625( proc.memory.availableMB‚ 1788.0546875# proc.memory.rssMB‚ 2725.75390625 disk.drbd1777.out‚0 disk.rbd2.in‚0( disk./.usagePercent‚74.3016183587959 disk.sda3.out‚ 179.78125! disk.dm-0.out‚43004.716796875) proc.memory.percent‚16.63668155670166# disk.nvme0c0n1.in‚ 1754.04296875& disk.nvme0c0n1.out‚43004.716796875 disk.rbd1.out‚ 2.3271484375! disk.rbd2.out‚443.47607421875 network.sent‚ 1.3389022e+07 disk.sda1.out‚0$ memory_percent‚89.08658027648926 proc.cpu.threads‚17 disk.rbd1.in‚0% disk./.usageGB‚1107.1478080749512 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda.out‚ 179.78125 disk.sda1.in‚0 disk.rbd3.in‚0Óä¥j2Ñ- _wandbruntime‚4754  _runtime‚4754ŸÑ%ÉÒà _step‚460 ' lr-AdamW/pg1‚0.00024895193778990576 ' lr-AdamW/pg2‚0.00024895193778990576 trainer/global_step‚5649 % _timestamp‚1.7884423145328896e+09 _runtime‚4754.577629656 DJÎÓÈ trainer/global_step‚5649 % _timestamp‚1.7884423145328896e+09 _runtime‚4754.577629656 _step‚460 ' lr-AdamW/pg1‚0.00024895193778990576 ' lr-AdamW/pg2‚0.00024895193778990576ÌO}µ2Ô- _wandbruntime‚4754  _runtime‚4754"L:[ (ÕƒP TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39144179 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚628482066926592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638954362005504 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39143736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105241045 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10457572633608 $ _timestamp‚1.788442314533659e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚638956988606464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14365283596288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚461 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83186239 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14365327895552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10472321470464 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188427290 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83186245 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93102299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638956988606464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53958117 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105241045 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚537667064697856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _runtime‚4754.578511158 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93101853 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105221011 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537667036563456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53958120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552032392593408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628959169030152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚628482066926592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10472321470464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618501596396544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚628482040535040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10454974348940 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83186239 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10474921679872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚628482040535040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10454974348940 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83186239 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10474921679872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638954362005504 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39143736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105241045 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10457572633608 $ _timestamp‚1.788442314533659e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚638956988606464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14365283596288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚461 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83186239 [LDeviceStatsMonitor.on_train_batch_start/inactÑØUive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14365327895552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10472321470464 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188427290 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83186245 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93102299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638956988606464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53958117 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105241045 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚537667064697856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _runtime‚4754.578511158 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93101853 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105221011 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537667036563456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53958120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552032392593408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628959169030152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚628482066926592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10472321470464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618501596396544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10474153799168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83200475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53967313 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537759452751360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53967316 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10474153799168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639066760653824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537759424616960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚628590006631936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚628589980240384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10456803691608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639066760653824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552127235697152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39150934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10476754021888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83200475 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93118250 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10459401976484 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚462 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 _runtime‚4755.319757055 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93117822 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚629064715834456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618607912142848 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105239245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚628590006631936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚629067339547300 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚618607937570816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚628589980240384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552127308117504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425vÒì:'Û“N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552127235697152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39150934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10476754021888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83200475 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93118250 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10459401976484 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚462 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 _runtime‚4755.319757055 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93117822 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚629064715834456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618607912142848 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105239245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚628590006631936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚629067339547300 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚618607937570816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚628589980240384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚552127308117504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14367855366144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10474153799168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83200475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53967313 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚537759452751360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53967316 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10474153799168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639066760653824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537759424616960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚628590006631936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚628589980240384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10456803691608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639066760653824Ή2Ü- _wandbruntime‚4755  _runtime‚4755./¨\ÝÖ 2loss/SRqqt1/assignment_loss‚1.1172659458592533 0loss/SRqqt3/detection_loss‚2.636837687095006 0loss/FRt1/assignment_loss‚1.3356008206383656 .loss/FRt2/detection_loss‚2.611300999055738 /loss/FRt3/detection_loss‚2.7864430312600024 /loss/FBt2/detection_loss‚2.9781046509742737 /loss/FBt4/detection_loss‚1.9290956020355225 epoch‚4 % _timestamp‚1.7884423152761745e+09 _runtime‚4755.320717876 1loss/SRqqt4/assignment_loss‚1.273526996205328 2loss/SRqqt2/assignment_loss‚1.6722993557119654 0loss/FBt3/assignment_loss‚0.3224705779692158 /loss/FRt4/detection_loss‚2.4159613407176472 1loss/SRqqt2/detection_loss‚1.7442193056146305 /loss/FBt1/detection_loss‚0.6192984243624666 /loss/FBt3/detection_loss‚2.5318164110183714 2loss/SRqqt3/assignment_loss‚1.8775666708747545 1loss/SRqqt1/detection_loss‚1.0718647450829546 1loss/SRqqt4/detection_loss‚1.8648349925875665 trainer/global_step‚5649 0loss/FRt2/assignment_loss‚3.0919701495896215 0loss/FBt1/assignment_loss‚0.1909580246645618 1loss/FBt2/assignment_loss‚0.17701755669391292 %loss/total_loss‚42.37877402837863 _step‚463 /loss/FRt3/assignment_loss‚3.279565794127328 0loss/FRt4/assignment_loss‚2.7274785702643185 0loss/FBt4/assignment_loss‚0.3618268172838725 /loss/FRt1/detection_loss‚1.7614495586919583@8maÞÛ 2loss/SRqqt2/assignment_loss‚1.6722993557119654 0loss/FBt3/assignment_loss‚0.3224705779692158 /loss/FRt4/detection_loss‚2.4159613407176472 1loss/SRqqt2/detection_loss‚1.7442193056146305 /loss/FBt1/detection_loss‚0.6192984243624666 /loss/FBt3/detection_loss‚2.5318164110183714 2loss/SRqqt3/assignment_loss‚1.8775666708747545 1loss/SRqqt1/detection_loss‚1.0718647450829546 1loss/SRqqt4/detection_loss‚1.8648349925875665 trainer/global_step‚5649 0loss/FRt2/assignment_loss‚3.0919701495896215 0loss/FBt1/assignment_loss‚0.1909580246645618 1loss/FBt2/assignment_loss‚0.17701755669391292 %loss/total_loss‚42.37877402837863 _step‚463 /loss/FRt3/assignment_loss‚3.279565794127328 0loss/FRt4/assignment_loss‚2.7274785702643185 0loss/FBt4/assignment_loss‚0.3618268172838725 /loss/FRt1/detection_loss‚1.7614495586919583 2loss/SRqqt1/assignment_loss‚1.1172659458592533 0loss/SRqqt3/detection_loss‚2.636837687095006 0loss/FRt1/assignment_loss‚1.3356008206383656 .loss/FRt2/detection_loss‚2.611300999055738 /loss/FRt3/detection_loss‚2.7864430312600024 /loss/FBt2/detection_loss‚2.9781046509742737 /loss/FBt4/detection_loss‚1.9290956020355225 epoch‚4 % _timestamp‚1.7884423152761745e+09 _runtime‚4755.320717876 1loss/SRqqt4/assignment_loss‚1.273526996205328ÏíÉ &“ß‚(:ˆ ÙíåÔ¦õúÈ gpu.0.gpu‚60.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚251.915' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.383 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð*­VAà‚(:¶ ÙíåԵϑË( disk./.usagePercent‚74.3016347432781 disk.rbd2.in‚0# proc.memory.rssMB‚ 2725.75390625 network.sent‚ 1.3491854e+07 disk.drbd1777.in‚0 disk.sda3.in‚0.5625& disk.nvme0c0n1.out‚43005.666015625 disk.dm-1.in‚0.5625 cpu‚25.296322486682524 network.recv‚ 1.765117e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 2.39013671875 disk.sda3.out‚ 180.76953125! disk.rbd2.out‚443.87841796875 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1754.04296875 disk.dm-1.out‚ 180.76953125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda.out‚ 180.76953125 disk.sda1.out‚0$ disk.nvme0n1.out‚43005.666015625 disk.rbd0.out‚0( proc.memory.availableMB‚ 1788.0078125% disk./.usageGB‚1107.1480522155762 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875$ memory_percent‚89.08686637878418 disk.sda1.in‚0 disk.rbd3.in‚0! disk.dm-0.out‚43005.666015625 disk.rbd0.in‚0) proc.memory.percent‚16.63668155670166³ÎVÅžá‚(:“ èíåÔÓÒŠÉ gpu.0.gpu‚73.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚97.514' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.502799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø5«â‚(:° èíåÔáŸÁÊ! disk.rbd2.out‚443.88623046875 disk.sda2.out‚0( disk./.usagePercent‚74.3016347432781 disk.rbd0.in‚0! disk.dm-0.out‚43006.775390625 disk.rbd3.in‚0) proc.memory.percent‚16.63668155670166& disk.nvme0c0n1.out‚43006.775390625 disk.sda.out‚ 181.140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 181.140625 disk.drbd1777.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚43006.775390625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 network.recv‚ 1.771329e+06% disk./.usageGB‚1107.1480522155762# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-1.out‚ 181.140625 disk.rbd1.out‚ 2.39013671875 network.sent‚ 1.3589932e+07 disk.sda3.in‚0.5625 disk.rbd3.out‚0$ memory_percent‚89.08541202545166 cpu‚25.29561344223481 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625# proc.memory.rssMB‚ 2725¹Q§/.75390625! disk.nvme0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1788.24609375³œkF2ã- _wandbruntime‚4794  _runtime‚4794ʘ·ÇäÁ & lr-AdamW/pg1‚0.0002462650677475798 & lr-AdamW/pg2‚0.0002462650677475798 trainer/global_step‚5699 % _timestamp‚1.7884423539837902e+09 _runtime‚4794.028445984 _step‚464:ÔÝÌåÆ & lr-AdamW/pg1‚0.0002462650677475798 & lr-AdamW/pg2‚0.0002462650677475798 trainer/global_step‚5699 % _timestamp‚1.7884423539837902e+09 _runtime‚4794.028445984 _step‚464Ð/À™2æ- _wandbruntime‚4794  _runtime‚4794îcI (çƒP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39482289 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623818629682176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10546455073548 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10566552528896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5699 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106156097 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚633879025802240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚634365084755724 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83898045 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93900696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14491187089920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83898039 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚644445604722688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10563952319488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54417917 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39482776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542286467373056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542286439238656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644442978121728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623818655110144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644442978121728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚644445604722688_dí(èˆP F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190034102 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83898039 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 _step‚465 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106136063 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106136063 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556777654462976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93900696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14491187089920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83898039 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚644445604722688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10563952319488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54417917 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39482776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542286467373056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542286439238656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚644442978121728 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623818655110144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644442978121728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚644445604722688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39482289 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623818629682176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10546455073548 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10566552528896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5699 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106156097 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚633879025802240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚634365084755724 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83898045 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚633986965507584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83912281 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644555376770048 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39489370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39488902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106154297 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83912275 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14493694635520 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106174357 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10565784648192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623924970856448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106174357 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623924996284416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚633986965507584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚542378855426560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10548284416216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚633986991899136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚634473255272664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10568384870912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542378827292160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106154297 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93916015 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10568384870912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609"6*FÖë“N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93916015 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10568384870912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚633986965507584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83912281 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644555376770048 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39489370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39488902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106154297 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83912275 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14493694635520 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106174357 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10565784648192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623924970856448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106174357 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623924996284416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚633986965507584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚542378855426560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10548284416216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚633986991899136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚634473255272664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10568384870912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542378827292160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106154297 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÒV0æý2ì- _wandbruntime‚4794  _runtime‚4794[˜ž>^íØ .loss/FRt2/assignment_loss‚2.95434981363791 2loss/SRqqt1/assignment_loss‚1.1043423464391053 2loss/SRqqt2/assignment_loss‚1.2525675807227479 trainer/global_step‚5699 _step‚467 1loss/SRqqt2/detection_loss‚1.9284249344404707 /loss/FBt3/detection_loss‚2.0057532668113707 _runtime‚4794.772311354 1loss/FBt1/assignment_loss‚0.08597744242093736 2loss/FBt4/assignment_loss‚0.053161983067790665 0loss/FRt3/assignment_loss‚3.2220988418997787 /loss/FRt2/detection_loss‚2.1803853738087193 .loss/FRt3/detection_loss‚2.257150033266523 .loss/FBt1/detection_loss‚0.573834189743993 /loss/FBt2/detection_loss‚6.3856507539749146 epoch‚4 /loss/FRt4/assignment_loss‚3.149144111644654 -loss/FRt4/detection_loss‚2.54887611809231 1loss/SRqqt4/detection_loss‚1.9602823195358117 2loss/SRqqt3/assignment_loss‚1.6007391127447288 1loss/SRqqt4/assignment_loss‚1.311313445524623 3loss/FBt2/assignment_loss‚0.0030898945406079292 1loss/FBt3/assignment_loss‚0.12184618534520268 /loss/FRt1/detection_loss‚1.5708869052480678 1loss/SRqqt1/detection_loss‚1.0111228738045483 1loss/SRqqt3/detection_loss‚2.4268117795387902 $ _timestamp‚1.788442354727752e+09 0loss/FRt1/assignment_loss‚1.4597415133398406 /loss/FBt4/detection_loss‚3.5917025009791055 &loss/total_loss‚44.759253320572554>ˆø{cîÝ 2loss/FBt4/assignment_loss‚0.053161983067790665 0loss/FRt3/assignment_loss‚3.2220988418997787 /loss/FRt2/detection_loss‚2.1803853738087193 .loss/FRt3/detection_loss‚2.257150033266523 .loss/FBt1/detection_loss‚0.573834189743993 /loss/FBt2/detection_loss‚6.3856507539749146 epoch‚4 /loss/FRt4/assignment_loss‚3.149144111644654 -loss/FRt4/detection_loss‚2.54887611809231 1loss/SRqqt4/detection_loss‚1.9602823195358117 2loss/SRqqt3/assignment_loss‚1.6007391127447288 1loss/SRqqt4/assignment_loss‚1.311313445524623 3loss/FBt2/assignment_loss‚0.0030898945406079292 1loss/FBt3/assignment_loss‚0.12184618534520268 /loss/FRt1/detection_loss‚1.5708869052480678 1loss/SRqqt1/detection_loss‚1.0111228738045483 1loss/SRqqt3/detection_loss‚2.4268117795387902 $ _timestamp‚1.788442354727752e+09 0loss/FRt1/assignment_loss‚1.4597415133398406 /loss/FBt4/detection_loss‚3.5917025009791055 &loss/total_loss‚44.759253320572554 .loss/FRt2/assignment_loss‚2.95434981363791 2loss/SRqqt1/assignment_loss‚1.1043423464391053 2loss/SRqqt2/assignment_loss‚1.2525675807227479 trainer/global_step‚5699 _step‚467 1loss/SRqqt2/detection_loss‚1.9284249344404707 /loss/FBt3/detection_loss‚2.0057532668113707 _runtime‚4794.772311354 1loss/FBt1/assignment_loss‚0.08597744242093736ÓXDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649931594237952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107071149 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚639776247906568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚639276037460992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚639276011069440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 % _timestamp‚1.7884423935397587e+09 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚639276037460992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84609845 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14616591862784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107051115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107071149 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39818360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10655583168512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94696080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54877720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚639276011069440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649931594237952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚561522385283072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624j«ŒõÚú‰P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84609839 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649934220838912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚561522461911040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94695606 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚639773624193932 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚629135713823744 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546905870048256 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚639776247906568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚639276037460992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚639276011069440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 % _timestamp‚1.7884423935397587e+09 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚639276037460992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84609845 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14616591862784 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107051115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107071149 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39818360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10655583168512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94696080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54877720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚639276011069440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649931594237952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191680994 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚561522385283072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649934220838912 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649931594237952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107071149 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84624081 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14619066657280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39824996 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39824563 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94711476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚561617373239296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚639383950774784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10657415497216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚639383950774784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54886916 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546998258101760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚639383977166336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639884418423716 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650041366272000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10642363425700 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 _step‚470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10657415497216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107069349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107089409 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107069349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10660015719936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54886913 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14619115137536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 trainer/global_step‚5749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84624075 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚629242054998016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650043992886272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚561617296624640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 _runtime‚4834.333363064:¤³\'ý“N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94711476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚561617373239296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191693424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚639383950774784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10657415497216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚639383950774784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54886916 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546998258101760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚639383977166336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639884418423716 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650041366272000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10642363425700 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 _step‚470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10657415497216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107069349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107089409 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107069349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10660015719936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54886913 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14619115137536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 trainer/global_step‚5749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84624075 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚629242054998016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚650043992886272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚561617296624640 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 _runtime‚4834.333363064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546998229967360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84624081 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650043992886272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84624081 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14619066657280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39824996 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39824563 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95492169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14742617443328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚645184787344776 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655422836955136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚6554202103541±ñFu76 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85321639 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655420210354176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644672996336640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884424325457706e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107966167 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10729416522764 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566267890166784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10747214017536 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55337517 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655422836955136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95492643 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10749814226944 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10749814226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10747214017536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚645182163632140 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10732014807432 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95492169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14742617443328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚645184787344776 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655422836955136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655420210354176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85321639 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655420210354176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644672996336640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884424325457706e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107966167 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10729416522764 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566267890166784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10747214017536 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55337517 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655422836955136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95492643 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10749814226944 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10749814226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10747214017536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚645182163632140 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10732014807432 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634559088283648 trainer/global_step‚5799 _step‚474 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚644780936041984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10733844150308 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10751646568960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40161896 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10749046346240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95508172 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚4873.325750682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655532609002496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85335875 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10751646568960 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85335875 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚566362788453888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 % _timestamp‚1.7884424332806726e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚644780962433536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634559113711616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55346713 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108004461 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚551617660776960!ülë“N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚644780962433536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚566362713936384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14745081293824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40161459 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10731245865432 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 % _timestamp‚1.7884424332806726e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚644780962433536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634559113711616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55346713 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108004461 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚551617660776960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634559088283648 trainer/global_step‚5799 _step‚474 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚644780936041984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10733844150308 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10751646568960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40161896 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10749046346240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95508172 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚4873.325750682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655532609002496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85335875 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10751646568960 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85335875 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚566362788453888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 FmƘÀ $ _timestamp‚1.788442471849861e+09 _runtime‚4911.894524211 _step‚476 & lr-AdamW/pg1‚0.0002381410531073901 & lr-AdamW/pg2‚0.0002381410531073901 trainer/global_step‚5849|›©Î˙Š_runtime‚4911.894524211 _step‚476 & lr-AdamW/pg1‚0.0002381410531073901 & lr-AdamW/pg2‚0.0002381410531073901 trainer/global_step‚5849 $ _timestamp‚1.788442471849861e+09ÜO<$2š- _wandbruntime‚4911  _runtime‚4911¡'\ (›ƒP QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108881219 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660911453071360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40490746 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10841445075968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96288063 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639769831250944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86033445 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14868482281984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86033439 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10841445075968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14868528678400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10820897247372 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108901253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10838844866560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚650593326782984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571013204077056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96288528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚639769805822976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚660908826470400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚650590703070348 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚5849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86033439 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660908826470400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194934698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556144675398656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108881219 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86033445 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108901253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚03¼là"œˆP PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86033445 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14868482281984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86033439 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10841445075968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14868528678400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10820897247372 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108901253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10838844866560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚650593326782984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571013204077056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96288528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚639769805822976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚660908826470400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚650590703070348 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚5849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86033439 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660908826470400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194934698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556144675398656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108881219 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86033445 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108901253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108881219 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660911453071360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40490746 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10841445075968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviší°^.ceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96288063 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639769831250944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248ÝÄÙö2- _wandbruntime‚4912  _runtime‚4912}¦»'žŽN FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _runtime‚4912.633343125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚478 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96303920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14871046606336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108919513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55806516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639876172425216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10825324874916 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86047675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10822726590040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96304369 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚650698873587288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639876146997248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86047681 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10843277417984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571108035540992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556237063452160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚650701497300132 trainer/global_step‚5849 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194947128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55806513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚650177947700736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚650177921309184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚650698873587288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639876146997248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86047681 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10843277417984 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571108035540992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556237063452160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚650701497300132 trainer/global_step‚5849 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194947128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55806513 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚650177947700736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚650177921309184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _runtime‚4912.633343125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚478 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96303920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14871046606336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108919513 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55806516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639876172425216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10825324874916 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86047675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10822726590040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574Þ€O 2 - _wandbruntime‚4912  _runtime‚4912eD^¡Ø 2loss/SRqqt4/assignment_loss‚1.4054019084108658 1loss/SRqqt1/detection_loss‚0.9699503460632903 0loss/SRqqt3/detection_loss‚2.141278620922204 _step‚479 epoch‚4 /loss/FRt2/assignment_loss‚2.532327703454278 0loss/FRt4/assignment_loss‚3.0622441534902536 /loss/FBt4/detection_loss‚3.2117850879828134 2loss/SRqqt2/assignment_loss‚1.3263625489508217 1loss/FBt4/assignment_loss‚0.16220935309926668 .loss/FRt1/detection_loss‚1.849992375312881 2loss/SRqqt1/assignment_loss‚0.9923518753743598 1loss/FBt1/assignment_loss‚0.12777303888404198 /loss/FBt1/detection_loss‚0.6751055556398224 trainer/global_step‚5849 % _timestamp‚1.7884424725898926e+09 0loss/FRt1/assignment_loss‚1.0734390751882033 1loss/FBt2/assignment_loss‚0.31658602371559635 /loss/FRt3/detection_loss‚2.3753512782149895 1loss/SRqqt2/detection_loss‚1.8661142327684037 1loss/SRqqt4/detection_loss‚1.6987657722879628 /loss/FBt3/detection_loss‚2.4259849105562483 _runtime‚4912.634455621 0loss/FBt3/assignment_loss‚0.1013890064620812 /loss/FRt4/detection_loss‚2.3603420689994214 %loss/total_loss‚41.23299705363219 0loss/FRt3/assignment_loss‚3.9352633588843875 2loss/SRqqt3/assignment_loss‚1.8197570749078735 .loss/FRt2/detection_loss‚2.381724940104918 /loss/FBt2/detection_loss‚2.4214967439572015éb ^c¢Ý 0loss/FRt1/assignment_loss‚1.0734390751882033 1loss/FBt2/assignment_loss‚0.31658602371559635 /loss/FRt3/detection_loss‚2.3753512782149895 1loss/SRqqt2/detection_loss‚1.8661142327684037 1loss/SRqqt4/detection_loss‚1.6987657722879628 /loss/FBt3/detection_loss‚2.4259849105562483 _runtime‚4912.634455621 0loss/FBt3/assignment_loss‚0.1013890064620812 /loss/FRt4/detection_loss‚2.3603420689994214 %loss/total_loss‚41.23299705363219 0loss/FRt3/assignment_loss‚3.9352633588843875 2loss/SRqqt3/assignment_loss‚1.8197570749078735 .loss/FRt2/detection_loss‚2.381724940104918 /loss/FBt2/detection_loss‚2.4214967439572015 2loss/SRqqt4/assignment_loss‚1.4054019084108658 1loss/SRqqt1/detection_loss‚0.9699503460632903 0loss/SRqqt3/detection_loss‚2.141278620922204 _step‚479 epoch‚4 /loss/FRt2/assignment_loss‚2.532327703454278 0loss/FRt4/assignment_loss‚3.0622441534902536 /loss/FBt4/detection_loss‚3.2117850879828134 2loss/SRqqt2/assignment_loss‚1.3263625489508217 1loss/FBt4/assignment_loss‚0.16220935309926668 .loss/FRt1/detection_loss‚1.849992375312881 2loss/SRqqt1/assignment_loss‚0.9923518753743598 1loss/FBt1/assignment_loss‚0.12777303888404198 /loss/FBt1/detection_loss‚0.6751055556398224 trainer/global_step‚5849 % _timestamp‚1.7884424725898926e+09ßr!•“£‚(:ˆ ïîåÔäü¥É gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚260.484' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·dª&3¤‚(:¨ ïîåÔôõºË% disk./.usageGB‚1107.1043319702148 disk.rbd1.in‚0 disk.sda2.in‚0 network.recv‚ 1.815921e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1787.609375 network.sent‚ 1.3951486e+07! disk.rbd2.out‚445.94580078125 disk.sda.out‚ 185.328125$ memory_percent‚89.08929824829102 disk.sda3.in‚0.5625 disk.sda3.out‚ 185.328125 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.rbd1.out‚2.453125 disk.sda1.in‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚43139.44921875% disk.nvme0c0n1.out‚43139.44921875 cpu‚25.288849970877386 disk.dm-0.out‚43139.44921875 disk.dm-1.out‚ 185.328125 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 proc.cpu.threads‚17 disk.dm-0.in‚ 1754.04296875 disk.drbd1777.in‚0) disk./.usagePercent‚74.29870064092817 disk.rbd2.in‚0 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875ƒ+kò”¥‚(:‰ þîåÔ–£ýÈ gpu.0.gpu‚73.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚153.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#MøËB¦‚(:· þîåÔ÷À«Ë) proc.memory.percent‚16.63668155670166 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625) disk./.usagePercent‚74.30699835712798 disk.rbd1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-0.in‚ 1754.04296875# disk.dm-0.out‚43266.78173828125 network.recv‚ 1.821726e+06 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚43266.78173828125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.75390625% disk./.usageGB‚1107.2279739379883 disk.rbd1.out‚2.453125 disk.sda3.out‚ 186.05859375 cpu‚25.288078909359182 disk.sda.out‚ 186.05859375( proc.memory.availableMB‚ 1787.4609375 disk.drbd1777.in‚0# memory_percent‚89.0902042388916 network.sent‚ 1.4059534e+07! disk.nvme0n1.in‚ 1754.04296875 disk.dm-1.out‚ 186.05859375 proc.cpu.threads‚17! disk.rbd2.out‚447.52001953125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚43266.78173828125Œ˜=a“§‚(:ˆ ïåÔêâòÈ gpu.0.gpu‚69.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚96.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.3998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡CÏÿA¨‚(:¶ ïåÔÀÍéÊ disk.sda2.out‚0# proc.memory.rssMB‚ 2725.75390625 disk.dm-0.in‚ 1754.04296875 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 186.76953125 disk.sda1.out‚0 disk.sda3.out‚ 186.76953125 disk.rbd0.in‚0& proc.memory.availableMB‚ 1787.46875 network.recv‚ 1.826066e+06) disk./.usagePercent‚74.30699835712798% disk./.usageGB‚1107.2279739379883 disk.sda2.in‚0& disk.nvme0n1.out‚43267.09814453125$ memory_percent‚89.09015655517578 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1754.04296875# disk.nvme0c0n1.in‚ 1754.04296875 disk.dm-1.in‚0.5625 network.sent‚ 1.4065525e+07 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.rbd3.out‚0# disk.dm-0.out‚43267.09814453125 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚43267.09814453125 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 186.76953125) proc.memory.percent‚16.63668155670166 cpu‚25.287416967909518! disk.rbd2.out‚447.62548828125 disk.sda1.in‚0BÐj2©- _wandbruntime‚4951  _runtime‚4951ø¢‡Gɪà ' lr-AdamW/pg1‚0.00023541379454985187 ' lr-AdamW/pg2‚0.00023541379454985187 trainer/global_step‚5899 % _timestamp‚1.7884425111027625e+09 _runtime‚4951.147420633 _step‚480Ë{ŒéÎ«È % _timestamp‚1.7884425111027625e+09 _runtime‚4951.147420633 _step‚480 ' lr-AdamW/pg1‚0.00023541379454985187 ' lr-AdamW/pg2‚0.00023541379454985187 trainer/global_step‚5899àÞI*µ2¬- _wandbruntime‚4951  _runtime‚4951eË— ­P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚655466966871040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚655466993262592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 E?DeviceStatsMonitor.o¯M6pn_train_batch_start/oversize_segments.freed‚0 # _timestamp‚1.78844251110352e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10930475715584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10930475715584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚560764078073856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14994266255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚655466966871040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56257120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666397442586624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655999242508556 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666400069187584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10933075924992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86745245 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97082792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56257117 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40826117 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86745245 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40825675 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109816305 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196561550 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚655466993262592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86745239 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109796271 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575758269798912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666397442586624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376mk? (®†P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 trainer/global_step‚5899 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109796271 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚655466993262592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 # _timestamp‚1.78844251110352e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10930475715584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10930475715584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚560764078073856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14994266255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚655466966871040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56257120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666397442586624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655999242508556 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666400069187584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10933075924992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86745245 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97082792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56257117 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40826117 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86745245 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40825675 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109816305 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196561550 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚655466993262592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86745239 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109796271 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575758269798912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666397442586624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚655466966871040 J³ÿ2¯- _wandbruntime‚4951  _runtime‚4951F-–¯'°ŽN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10934908267008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚666509841234944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560856437992960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10934908267008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109814505 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚560856466127360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚5899 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109814505 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚655574932967936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86759475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56266313 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚666507214620672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656110036738340 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86759481 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97098592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14996726436352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4951.903966203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56266316 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86759475 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86759481 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40832707 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14996774916608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10916805599524 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚655574906576384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚666509841234944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97099023 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚656107413025496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884425118588006e+09-Ž±“N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10932308044288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109834565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚575853241043968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109814505 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚655574932967936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86759475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56266313 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚666507214620672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656110036738340 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86759481 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97098592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14996726436352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4951.903966203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56266316 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86759475 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86759481 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40832707 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 RBDeñX„þviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14996774916608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10916805599524 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚655574906576384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚666509841234944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97099023 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚656107413025496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884425118588006e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10934908267008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚666509841234944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560856437992960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10934908267008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109814505 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚560856466127360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚5899 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661407781946764 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671886058702848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11006456981256 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11024706774016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97880243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660863978529792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56716917 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 trainer/global_step‚5949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402^pŠM(¾‰P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198168362 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580503105745408 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11024706774016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97879767 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56716920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚671886058702848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110731357 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97880243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚660863978529792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56716917 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 trainer/global_step‚5949 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87457045 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11003858696588 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671888685303808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661407781946764 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671886058702848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11006456981256 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11024706774016 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97895555 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661515952463704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41169442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565475868802560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661518576176548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671998457351168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11026539116032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660971918235136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56726113 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87471275 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580597912014848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11023938893312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11026539116032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11008286324132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97896011 trainer/global_step‚5949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660971891843584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660971891843584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87471281 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660971918235136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110749617 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41169895 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110749617 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671998457351168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580597988629504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚671995830736896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884425512308948e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110729557 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87471275 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110729557 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚650510264424448åšÖ,'Á“N FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚660971918235136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56726113 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87471275 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580597912014848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11023938893312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11026539116032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11008286324132 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97896011 trainer/global_step‚5949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660971891843584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660971891843584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87471281 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660971918235136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110749617 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41169895 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110749617 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671998457351168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198200832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚580597988629504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚671995830736896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884425512308948e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110729557 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87471275 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110729557 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚650510264424448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97895555 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚661515952463704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41169442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565475868802560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661518576176548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671998457351168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11026539116032æeM#ÿ2Â- _wandbruntime‚4991  _runtime‚4991,øs'[ÃÕ -loss/FRt4/detection_loss‚2.35729494205741 /loss/FBt3/detection_loss‚2.4654764262112705 /loss/FBt4/detection_loss‚2.9126586516698203 trainer/global_step‚5949 /loss/FRt3/assignment_loss‚3.113969016075134 0loss/FRt4/assignment_loss‚2.7062021729558015 epoch‚4 _step‚487 0loss/FBt2/assignment_loss‚0.2805493690939329 % _timestamp‚1.7884425512329533e+09 0loss/FRt2/assignment_loss‚3.2230985973562514 .loss/FRt1/detection_loss‚1.701281854566538 1loss/SRqqt1/detection_loss‚0.9088255884446974 2loss/SRqqt1/assignment_loss‚1.0747474838080335 0loss/FBt3/assignment_loss‚0.2649823297820562 /loss/FRt3/detection_loss‚2.2849525538503106 1loss/SRqqt3/detection_loss‚2.2638863836015974 1loss/SRqqt4/detection_loss‚1.7060267162525047 /loss/FBt1/detection_loss‚0.5221490801535159 2loss/SRqqt3/assignment_loss‚1.8483426005446486 1loss/FBt1/assignment_loss‚0.05604886932629253 1loss/SRqqt2/detection_loss‚2.2000805957644594 /loss/FBt2/detection_loss‚2.6670589596033096 0loss/FRt1/assignment_loss‚1.5596545541061546 2loss/SRqqt4/assignment_loss‚1.2765009712617277 1loss/FBt4/assignment_loss‚0.08958861806119482 /loss/FRt2/detection_loss‚2.5739830008574893 %loss/total_loss‚41.37676254330156 _runtime‚ 4991.2774938 2loss/SRqqt2/assignment_loss‚1.3194032078974096Ä·Ý`ÄÚ 0loss/FRt2/assignment_loss‚3.2230985973562514 .loss/FRt1/detection_loss‚1.701281854566538 1loss/SRqqt1/detection_loss‚0.9088255884446974 2loss/SRqqt1/assignment_loss‚1.0747474838080335 0loss/FBt3/assignment_loss‚0.2649823297820562 /loss/FRt3/detection_loss‚2.2849525538503106 1loss/SRqqt3/detection_loss‚2.2638863836015974 1loss/SRqqt4/detection_loss‚1.7060267162525047 /loss/FBt1/detection_loss‚0.5221490801535159 2loss/SRqqt3/assignment_loss‚1.8483426005446486 1loss/FBt1/assignment_loss‚0.05604886932629253 1loss/SRqqt2/detection_loss‚2.2000805957644594 /loss/FBt2/detection_loss‚2.6670589596033096 0loss/FRt1/assignment_loss‚1.5596545541061546 2loss/SRqqt4/assignment_loss‚1.2765009712617277 1loss/FBt4/assignment_loss‚0.08958861806119482 /loss/FRt2/detection_loss‚2.5739830008574893 %loss/total_loss‚41.37676254330156 _runtime‚ 4991.2774938 2loss/SRqqt2/assignment_loss‚1.3194032078974096 -loss/FRt4/detection_loss‚2.35729494205741 /loss/FBt3/detection_loss‚2.4654764262112705 /loss/FBt4/detection_loss‚2.9126586516698203 trainer/global_step‚5949 /loss/FRt3/assignment_loss‚3.113969016075134 0loss/FRt4/assignment_loss‚2.7062021729558015 epoch‚4 _step‚487 0loss/FBt2/assignment_loss‚0.2805493690939329 % _timestamp‚1.7884425512329533e+09çûé sžÅ‚(:“ ºïåÔÜ´É gpu.0.gpu‚84.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚259.176' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.83519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=æq 7Æ‚(:¬ ºïåÔÓïÏË( disk.nvme0c0n1.out‚43276.89794921875# disk.dm-0.out‚43276.89794921875 disk.drbd1777.out‚0$ memory_percent‚89.09072875976562$ proc.memory.availableMB‚1787.375 disk.sda3.in‚0.5625 disk.sda3.out‚ 188.296875) proc.memory.percent‚16.63668155670166 network.recv‚ 1.840925e+06 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875& disk.nvme0n1.out‚43276.89794921875# disk.nvme0c0n1.in‚ 1754.04296875 disk.drbd1777.in‚0 disk.dm-1.out‚ 188.296875 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2725.75390625) disk./.usagePercent‚74.30753008477659 disk.rbd1.out‚2.453125 disk.sda1.in‚0 disk.dm-0.in‚ 1754.04296875 disk.sda.in‚0.5625 disk.sda.out‚ 188.296875 disk.rbd3.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 network.sent‚ 1.4185646e+07$ disk./.usageGB‚1107.235897064209 disk.rbd1.in‚0 disk.rbd2.out‚450.8212890625 disk.sda1.out‚0 disk.dm-1.in‚0.5625 cpu‚25.285281099495272¢êe²”Ç‚(:‰ ÉïåÔº·ÏÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚248.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô×=È‚(:² ÉïåÔ÷²ÍË cpu‚25.284531135436342 network.recv‚ 1.847308e+06$ disk./.usageGB‚1107.235897064209 disk.sda3.in‚0.5625 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.04296875# memory_percent‚89.0902042388916( proc.memory.availableMB‚ 1787.4609375) disk./.usagePercent‚74.30753008477659 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1754.04296875# proc.memory.rssMB‚ 2725.75390625( disk.nvme0c0n1.out‚43277.23779296875 network.sent‚ 1.4294609e+07 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 188.7578125 disk.sda2.out‚0 disk.sda3.out‚ 188.7578125 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.out‚452.5009765625 disk.sda.in‚0.5625 disk.rbd3.out‚0& disk.nvme0n1.out‚43277.23779296875 disk.dm-0.in‚ 1754.04296875) proc.memory.percent‚16.63668155670166 disk.rbd1.out‚2.453125# disk.dm-0.out‚43277.23779296875 disk.dm-1.out‚ 188.7578125]‘öݞɂ(:“ ØïåÔ¡ÂôÈ gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚262.648' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.52960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 óñ8Ê‚(:­ ØïåÔÀ¹ÅÌ# proc.memory.rssMB‚ 2725.75390625) proc.memory.percent‚16.63668155670166 disk.sda.out‚ 189.328125 disk.sda2.in‚0% disk.nvme0n1.out‚43277.6064453125 disk.rbd0.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚43277.6064453125 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 189.328125 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.04296875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.30752445261083 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1787.33203125 disk.dm-0.in‚ 1754.04296875 disk.sda3.out‚ 189.328125$ memory_percent‚89.09099102020264 network.sent‚ 1.4300652e+07 network.recv‚ 1.851766e+06% disk./.usageGB‚1107.2358131408691 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.04296875 cpu‚25.2838744517834' disk.nvme0c0n1.out‚43277.6064453125 disk.rbd1.out‚2.453125 disk.rbd2.out‚452.5009765625 disk.sda.in‚0.5625 disk.sda1.in‚0H÷ñ}2Ë- _wandbruntime‚5029  _runtime‚5029L €ÉÌà ' lr-AdamW/pg2‚0.00022993380398927908 trainer/global_step‚5999 % _timestamp‚1.7884425895564802e+09 _runtime‚5029.601134142 _step‚488 ' lr-AdamW/pg1‚0.00022993380398927908¿ÁCÎÍÈ trainer/global_step‚5999 % _timestamp‚1.7884425895564802e+09 _runtime‚5029.601134142 _step‚488 ' lr-AdamW/pg1‚0.00022993380398927908 ' lr-AdamW/pg2‚0.00022993380398927908èÔnZ¢2Î- _wandbruntime‚5029  _runtime‚5029´åÏظσP O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41498593 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570002883424256 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788442589557201e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666260937405440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚677374674819072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570002855289856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111646409 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666260937405440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚5999 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11095339421196 MGDeviceStatsMonitor.on_train_"€ºQbatch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11116337623040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98674832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15245092744704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111626375 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666816321384972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11113737413632 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88168845 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 _step‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11116337623040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57176717 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57176720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚677374674819072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570002855289856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111646409 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666260937405440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚5999 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11095339421196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11116337623040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98674832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15245092744704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111626375 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666816321384972 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11113737413632 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88168845 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 _step‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11116337623040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57176717 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57176720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788442589557201e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666260937405440 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655827348566016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98690578 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111644609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57185916 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚677487073467392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111664669 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666924491901912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11099767048740 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111644609 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666368877110784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11115569742336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666368903502336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11118169965056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88183075 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 % _timestamp‚1.7884425902971213e+09 _step‚490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚666368903502336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11097168763864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199827684 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚655827323138048 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41505132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111664669 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11115569742336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98691048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5999 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88183081 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 _runtime‚ 5030.34217095 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11118169965056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666927115614756 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467£†ð9!Ó’N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚677487073467392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111664669 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666924491901912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11099767048740 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111644609 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666368877110784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11115569742336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666368903502336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11118169965056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88183075 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 % _timestamp‚1.7884425902971213e+09 _step‚490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚666368903502336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11097168763864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199827684 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚655827323138048 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41505132 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111664669 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11115569742336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98691048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5999 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88183081 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 _runtime‚ 5030.34217095 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11118169965056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666927115614756 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚655827348566016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98690578 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_b±ü{ßytes.all.peak‚ 20424465233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111644609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57185916 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ê_`bY2Ô- _wandbruntime‚5030  _runtime‚5030ÊØa^ÕØ 0loss/FRt4/assignment_loss‚3.0300496845018294 2loss/SRqqt2/assignment_loss‚1.2441915608197451 /loss/FRt1/detection_loss‚1.3937060386573865 /loss/FRt2/detection_loss‚2.3918056695357612 /loss/FBt2/detection_loss‚2.2930895388126373 %loss/total_loss‚41.30717426909099 /loss/FRt3/assignment_loss‚3.630693331360817 2loss/SRqqt3/assignment_loss‚1.4422536783386022 .loss/FRt4/detection_loss‚2.341904418809073 2loss/SRqqt4/assignment_loss‚1.7735080154652292 1loss/FBt1/assignment_loss‚0.18157955632231407 1loss/SRqqt2/detection_loss‚1.7594758108258248 /loss/FRt2/assignment_loss‚3.009627272253451 2loss/SRqqt1/assignment_loss‚0.8568369039752777 1loss/FBt2/assignment_loss‚0.35811746858234983 epoch‚4 % _timestamp‚1.7884425902990005e+09 1loss/FBt4/assignment_loss‚0.46201119509836036 /loss/FRt3/detection_loss‚2.3777802315173804 0loss/SRqqt3/detection_loss‚2.292127003893256 /loss/FBt1/detection_loss‚0.5015830358922347 0loss/FRt1/assignment_loss‚1.4823735558380515 0loss/FBt3/assignment_loss‚0.7188988062035706 1loss/SRqqt1/detection_loss‚0.8029981152410957 /loss/FBt3/detection_loss‚2.3260391685697765 _step‚491 1loss/SRqqt4/detection_loss‚1.7222981633658105 /loss/FBt4/detection_loss‚2.9142260452111564 trainer/global_step‚5999 _runtime‚5030.343611875ŽöfˆcÖÝ 1loss/FBt4/assignment_loss‚0.46201119509836036 /loss/FRt3/detection_loss‚2.3777802315173804 0loss/SRqqt3/detection_loss‚2.292127003893256 /loss/FBt1/detection_loss‚0.5015830358922347 0loss/FRt1/assignment_loss‚1.4823735558380515 0loss/FBt3/assignment_loss‚0.7188988062035706 1loss/SRqqt1/detection_loss‚0.8029981152410957 /loss/FBt3/detection_loss‚2.3260391685697765 _step‚491 1loss/SRqqt4/detection_loss‚1.7222981633658105 /loss/FBt4/detection_loss‚2.9142260452111564 trainer/global_step‚5999 _runtime‚5030.343611875 0loss/FRt4/assignment_loss‚3.0300496845018294 2loss/SRqqt2/assignment_loss‚1.2441915608197451 /loss/FRt1/detection_loss‚1.3937060386573865 /loss/FRt2/detection_loss‚2.3918056695357612 /loss/FBt2/detection_loss‚2.2930895388126373 %loss/total_loss‚41.30717426909099 /loss/FRt3/assignment_loss‚3.630693331360817 2loss/SRqqt3/assignment_loss‚1.4422536783386022 .loss/FRt4/detection_loss‚2.341904418809073 2loss/SRqqt4/assignment_loss‚1.7735080154652292 1loss/FBt1/assignment_loss‚0.18157955632231407 1loss/SRqqt2/detection_loss‚1.7594758108258248 /loss/FRt2/assignment_loss‚3.009627272253451 2loss/SRqqt1/assignment_loss‚0.8568369039752777 1loss/FBt2/assignment_loss‚0.35811746858234983 epoch‚4 % _timestamp‚1.7884425902990005e+09ë>„#|Ÿ×‚(:” çïåÔ€¦É gpu.0.gpu‚58.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚122.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.593799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WòH@Ø‚(:µ çïåÔ±‰Ë% disk./.usageGB‚1107.2358665466309 disk.dm-0.in‚ 1754.2109375 disk.rbd1.out‚2.453125 disk.sda2.in‚0& disk.nvme0n1.out‚43278.04443359375 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 1754.2109375( disk.nvme0c0n1.out‚43278.04443359375$ memory_percent‚89.08989429473877) proc.memory.percent‚16.63668155670166 disk.rbd2.out‚454.1767578125# disk.dm-0.out‚43278.04443359375 disk.drbd1777.out‚0 disk.sda3.in‚0.5625 network.sent‚ 1.4305901e+07 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.out‚ 189.85546875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.30752803671632 network.recv‚ 1.855467e+06 disk.rbd3.in‚0) proc.memory.availableMB‚ 1787.51171875# proc.memory.rssMB‚ 2725.75390625 cpu‚25.283147962864533 disk.dm-1.out‚ 189.85546875 disk.drbd1777.in‚0 disk.sda.out‚ 189.85546875 disk.sda1.in‚01ã¡žÙ‚(:“ öïåÔÿÓ×È gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨»¢7Ú‚(:¬ öïåÔššíË disk.dm-0.in‚ 1754.2109375 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda3.out‚ 190.34375 cpu‚25.282451874453333% disk./.usageGB‚1107.2357940673828( disk.nvme0c0n1.out‚43278.89892578125 disk.sda.in‚0.5625) proc.memory.percent‚16.63668155670166 network.recv‚ 1.861419e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.out‚0# disk.dm-0.out‚43278.89892578125 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2725.75390625) disk./.usagePercent‚74.30752317257317 disk.drbd1777.in‚0 disk.sda.out‚ 190.34375 network.sent‚ 1.4413949e+07 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚454.1767578125 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚43278.89892578125$ memory_percent‚89.08989429473877) proc.memory.availableMB‚ 1787.51171875" disk.nvme0c0n1.in‚ 1754.2109375 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 190.34375 disk.sda3.in‚0.5625Bó'6žÛ‚(:“ …ðåÔ­Ú…É gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž¼ð?Ü‚(:´ …ðåÔ¾Ž•Ë disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1754.2109375( disk.nvme0c0n1.out‚43279.35498046875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.out‚ 190.79296875 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.2109375 disk.drbd1777.in‚0$ memory_percent‚89.09087181091309# proc.memory.rssMB‚ 2725.75390625 disk.sda2.in‚0! disk.rbd2.out‚455.93505859375( proc.memory.availableMB‚ 1787.3515625) proc.memory.percent‚16.63668155670166 network.sent‚ 1.4419963e+07# disk.dm-0.out‚43279.35498046875 disk.rbd1.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚43279.35498046875 network.recv‚ 1.865665e+06$ disk./.usageGB‚1107.361228942871 disk.sda2.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.out‚ 190.79296875 disk.dm-1.in‚0.5625 disk.sda.out‚ 190.79296875 disk.sda1.in‚0) disk./.usagePercent‚74.31594121231406 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.rbd1.out‚2.453125 disk.sda1.out‚0 cpu‚25.285404374002308Ró‘Ý‚(:† ”ðåÔ̼ÿÈ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚37.0 gpu.0.powerWatts‚89.289' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í£Ú1Þ‚(:¦ ”ðåÔÆñ™Ì network.recv‚ 1.869711e+06 disk.dm-1.out‚ 191.21875 disk.sda3.out‚ 191.21875$ memory_percent‚89.09130096435547) proc.memory.percent‚16.63668155670166 cpu‚25.29859217517377% disk./.usageGB‚1107.3607063293457 disk.rbd2.in‚0! disk.rbd2.out‚455.93505859375 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.nvme0n1.in‚ 1754.2109375& proc.memory.availableMB‚ 1787.28125 disk.rbd1.out‚2.453125 disk.sda.out‚ 191.21875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.75390625 network.sent‚ 1.4425727e+07) disk./.usagePercent‚74.31590613928186 disk.dm-0.in‚ 1754.2109375" disk.dm-0.out‚43377.8623046875 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 1754.2109375' disk.nvme0c0n1.out‚43377.8623046875 disk.rbd1.in‚0% disk.nvme0n1.out‚43377.8623046875 disk.sda3.in‚0.5625L%m“ß‚(:ˆ £ðåÔ‹æ¯É gpu.0.gpu‚92.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚38.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ô•/9à‚(:® £ðåÔ©´„Ì disk.sda.in‚0.5625 disk.rbd3.in‚0 network.recv‚ 1.87428e+06) disk./.usagePercent‚74.31591381950788 disk.dm-0.in‚ 1754.2109375) proc.memory.percent‚16.63668155670166% disk.nvme0n1.out‚43378.1396484375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1754.2109375$ memory_percent‚89.08965587615967 disk.sda1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2725.75390625 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 191.8515625 network.sent‚ 1.4432019e+07 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.in‚0.5625 proc.cpu.threads‚17% disk./.usageGB‚1107.3608207702637" disk.dm-0.out‚43378.1396484375 disk.dm-1.out‚ 191.8515625! disk.rbd2.out‚457.67724609375 disk.sda.out‚ 191.8515625' disk.nvme0c0n1.out‚43378.1396484375) proc.memory.availableMB‚ 1787.55078125 cpu‚25.31200754110707 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0.5625éü-:œá‚(:‘ ²ðåÔ†öÿÈ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.821' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.56420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø :â‚(:¯ ²ðåÔݸöË disk.sda.out‚ 192.19921875) disk./.usagePercent‚74.32429550617891$ memory_percent‚89.09008502960205 disk.rbd2.in‚0! disk.rbd2.out‚459.25146484375# proc.memory.rssMB‚ 2725.75390625 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.out‚ 192.19921875 disk.rbd0.in‚0! disk.dm-0.out‚43378.724609375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 cpu‚25.32422608531332$ disk.nvme0n1.out‚43378.724609375) proc.memory.percent‚16.63668155670166 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0& disk.nvme0c0n1.out‚43378.724609375% disk./.usageGB‚1107.4857139587402 disk.rbd3.out‚0 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 1787.48046875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1754.2109375 disk.dm-0.in‚ 1754.2109375 disk.sda1.out‚0 network.sent‚ 1.4437222e+07 network.recv‚ 1.877828e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.dm-1.out‚ 192.19921875 proc.cpu.threads‚17úŠAª2ã- _wandbruntime‚5115  _runtime‚5115¡r½s*äíT eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚15384343196672 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 112824364 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚89357507 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 112840338 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23670554624 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3300917248 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚674840361050112 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚42007616 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 trainer/global_step‚6044 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚664042755347456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚686064246732288 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚89357502 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚686064246732288 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚57988306 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚592674194800640 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2648 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975012864 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15384464464384 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950718464 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚99996053 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚592674333716992 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1074 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 112840338 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11223885682176 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11205274873132 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚577289851603968 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 202181866 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1574 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚686062271719424 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚131 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2648 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚492 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚89357507 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15974 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚675246058087124 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23670554624 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 112824364 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚57988304 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 20369637376 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚674840361050112 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11221934963712 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1574 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚686062271719424 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚99995920 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 138916352xô*!åòT [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950718464 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚99996053 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚592674333716992 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1074 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 112840338 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11223885682176 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11205274873132 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚577289851603968 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 202181866 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1574 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚686062271719424 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚131 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2648 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚492 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚89357507 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15974 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚675246058087124 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23670554624 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 112824364 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚57988304 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 20369637376 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚674840361050112 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11221934963712 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1574 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚686062271719424 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚99995920 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 138916352 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚15384343196672 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 112824364 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚89357507 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 112840338 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23670554624 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_œ(±di pool.current‚ 3300917248 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚674840361050112 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚42007616 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 trainer/global_step‚6044 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚664042755347456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚686064246732288 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚89357502 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚686064246732288 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚57988306 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚592674194800640 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2648 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975012864 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15384464464384 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074ìJC2æ- _wandbruntime‚5116  _runtime‚5116®e€)çùR D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2648 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 138916352 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11203798756372 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 100008116 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚42011880 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11205747559020 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚134 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚686136504510464 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 112847552 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚57996234 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚592736790274560 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2648 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788341248 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 1975012864 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚674914119638016 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚42011748 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23670554624 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚89368401 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23670554624 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 trainer/global_step‚6044 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚674914095343616 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚57996236 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚2648 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚686138479523328 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202199974 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚89368401 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚577351833958400bí„)èþR % _timestamp‚1.7884426768807402e+09 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚15384817399808 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 100007982 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚592736651358208 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1574 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1574 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202199974 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚686138479523328 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1574 _runtime‚5116.925811856 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚686136504510464 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 112831578 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948802648 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3300917248 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚89368396 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11224359885312 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 112847552 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚674914119638016 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 121267712 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚664112756448256 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11222409166848 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚42011748 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23670554624 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚89368401 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23670554624 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 trainer/global_step‚6044 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚674914095343616 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚57996236 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚2648 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚686138479523328 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202199974 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚89368401 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚577351833958400 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2648 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 138916352 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11203798756372 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 100008116 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚42011880 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11205747559020 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚134 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚686136504510464 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 112847552 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚57996234 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚592736790274560 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2648 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788341248 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 1975012864 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚674914119638016 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5642856955528259 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5181037783622742 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2231762409210205 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015798576176166534 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49184420704841614 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971083402633667 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49670252203941345 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47861069440841675 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0829218253493309 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835516214370728 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3148147761821747 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17179101705551147 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.42519161105155945 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029746446758508682 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.$ªZWw5342113375663757 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 +jet/accuracy_2_of_2‚0.10982856899499893 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5056223273277283 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02600749582052231 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.42698395252227783 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5246003270149231 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38437411189079285 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2660636603832245 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4827655255794525 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16403509676456451 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5147680640220642 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3310869336128235 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_5_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910905957221985 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 *jet/accuracy_3_of_5‚0.0963197648525238 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.43300655484199524 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9881004691123962 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 _runtime‚5130.396521908 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994879961013794 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5692397356033325 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25596514344215393 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.138888880610466 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15123456716537476 6validation_average_jet_accuracy‚0.4395509362220764 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4122872054576874 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_4‚0.910903811454773 *jet/accuracy_3_of_6‚0.1538461595773697 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.988496720790863 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9957482814788818 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4236849546432495 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.515860915184021 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19770114123821259 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994879961013794 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 jet/accuracy_4_of_7‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15650269389152527 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43656131625175476 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47873398661613464 6"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5180624127388 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.982475996017456 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 particle/accuracy_7_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5050670504570007 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9685291647911072 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49562782049179077 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5921541452407837 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.181142196059227 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.037540942430496216 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5555555820465088 particle/accuracy_2_of_3‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1835632473230362 'particle/f_score‚0.3966263234615326 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5699884295463562 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48722925782203674 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 epoch‚4 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1877370923757553 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2659422755241394 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5050670504570007 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9901541471481323 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06024220213294029 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21630118787288666 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26088669896125793 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03892729431390762 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4439571797847748 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5105076432228088 *jet/accuracy_2_of_3‚0.2529905438423157 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02533692866563797 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41447368264198303 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5001419186592102 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20053166151046753 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_4‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1754467785358429 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18817918002605438 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,jet/accuracy_3_of_3‚0.027425043284893036 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41649138927459717 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15101826190948486 particle/accuracy_2_of_4‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14925578236579895 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32351189851760864 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13973921537399292 jet/accuracy_1_of_7‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9969051480293274 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988006591796875 ,jet/accuracy_4_of_4‚0.009997246786952019 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1743486225605011 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2750369906425476 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2515765428543091 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0300508514046669 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40573039650917053 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28238800168037415 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5389685034751892 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36918607354164124 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.686263918876648 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882731437683105 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1885521560907364 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4879811406135559 ,particle/sensitivity‚0.31385374069213867 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4815128743648529 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 _step‚494 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14222222566604614 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 *loss/val_total_loss‚42.087437363762234 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17569352686405182 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5218254327774048 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.975933849811554 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6127975583076477 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010582011193037033 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4777267277240753 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2606479227542877 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37678366899490356 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1534503698348999 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5603448152542114 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15095627307891846 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11640697717666626 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3097001910209656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2916666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861640930176 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_7_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981389045715332 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.37666165828704834 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15856440365314484 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19827644526958466 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07410714775323868 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5614582896232605 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9934649467468262 +particle/specificity‚0.9504374861717224 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚6044 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5292574763298035 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6476585268974304 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5033667087554932 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4880123734474182 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49154549837112427 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.53125 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022722799330949783 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4250841438770294 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8510199189186096 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 *jet/accuracy_2_of_6‚0.4743589758872986 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0702066645026207 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3125000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.982292652130127 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30868053436279297 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.013565892353653908 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 *jet/accuracy_1_of_3‚0.8711703419685364 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9773105382919312 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5339528322219849 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980139136314392 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17392666637897491 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25520816445350647 particle/accuracy_6_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7602505087852478 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.048489850014448166 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5765849947929382 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.986258864402771 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5084570646286011 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.017626726999878883 particle/accuracy_3_of_4‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.42189541459083557 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5271889567375183 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18063056468963623 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22327584028244019 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.34918150305747986 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9873362183570862 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22357596457004547 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5133198499679565 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 particle/accuracy_5_of_7‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09921371936798096 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22359523177146912 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3577539026737213 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.03846153989434242 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19212962687015533 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32833802700042725 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4771505892276764 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_6_of_6‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_2_of_5‚0.43283408880233765 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9899540543556213 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5373750329017639 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.460919588804245 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.25303030014038086 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5336787104606628 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996128678321838 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22446942329406738 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5347123742103577 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184R›íø¨ ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1743486225605011 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2750369906425476 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2515765428543091 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0300508514046669 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.40573039650917053 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28238800168037415 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5389685034751892 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36918607354164124 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.686263918876648 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882731437683105 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1885521560907364 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4879811406135559 ,particle/sensitivity‚0.31385374069213867 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4815128743648529 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 _step‚:“£Iù494 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14222222566604614 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 *loss/val_total_loss‚42.087437363762234 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17569352686405182 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5218254327774048 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.975933849811554 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6127975583076477 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.010582011193037033 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4777267277240753 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2606479227542877 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37678366899490356 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1534503698348999 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5603448152542114 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15095627307891846 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11640697717666626 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3097001910209656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2916666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861640930176 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_7_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981389045715332 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.37666165828704834 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15856440365314484 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19827644526958466 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07410714775323868 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5614582896232605 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9934649467468262 +particle/specificity‚0.9504374861717224 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚6044 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5292574763298035 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6476585268974304 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5033667087554932 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4880123734474182 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49154549837112427 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.53125 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022722799330949783 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4250841438770294 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8510199189186096 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 *jet/accuracy_2_of_6‚0.4743589758872986 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0702066645026207 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 jet/accuracy_6_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3125000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.982292652130127 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30868053436279297 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.013565892353653908 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 *jet/accuracy_1_of_3‚0.8711703419685364 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9773105382919312 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5339528322219849 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980139136314392 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17392666637897491 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25520816445350647 particle/accuracy_6_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7602505087852478 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.048489850014448166 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5765849947929382 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.986258864402771 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5084570646286011 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.017626726999878883 particle/accuracy_3_of_4‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.42189541459083557 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5271889567375183 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18063056468963623 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22327584028244019 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.34918150305747986 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9873362183570862 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22357596457004547 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5133198499679565 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 particle/accuracy_5_of_7‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09921371936798096 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22359523177146912 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3577539026737213 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.03846153989434242 particle/accuracy_2_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.019736841320991516 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19212962687015533 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32833802700042725 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4771505892276764 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_6_of_6‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_2_of_5‚0.43283408880233765 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9899540543556213 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5373750329017639 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.460919588804245 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.25303030014038086 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5336787104606628 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996128678321838 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22446942329406738 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5347123742103577 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20235289633274078 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5103922486305237 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9954648613929749 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16331884264945984 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012226012535393238 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5355556607246399 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18063056468963623 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19988290965557098 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2519700825214386 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20601625740528107 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4882853925228119 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17011763155460358 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9894524216651917 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49646294116973877 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.991550862789154 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 +jet/accuracy_3_of_4‚0.07246976345777512 particle/accuracy_2_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4986133277416229 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09923671931028366 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17254820466041565 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5257275700569153 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3242434859275818 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06085299700498581 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_1_of_4‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 % _timestamp‚1.7884426903497882e+09 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.42267265915870667 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3872847557067871 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889923334121704 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17979812622070312 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00561766279861331 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32592591643333435 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.285463809967041 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8169148564338684 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01616876944899559 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2613341510295868 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5676307678222656 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979703426361084 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19860316812992096 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5642856955528259 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5181037783622742 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2231762409210205 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015798576176166534 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.49184420704841614 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971083402633667 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49670252203941345 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47861069440841675 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0829218253493309 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835516214370728 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3148147761821747 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17179101705551147 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.42519161105155945 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029746446758508682 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5342113375663757 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 +jet/accuracy_2_of_2‚0.10982856899499893 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5056223273277283 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02600749582052231 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.42698395252227783 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5246003270149231 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38437411189079285 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2660636603832245 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4827655255794525 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16403509676456451 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5147680640220642 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3310869336128235 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_5_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910905957221985 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 *jet/accuracy_3_of_5‚0.0963197648525238 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.43300655484199524 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9881004691123962 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 _runtime‚5130.396521908 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994879961013794 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5692397356033325 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25596514344215393 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.138888880610466 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15123456716537476 6validation_average_jet_accuracy‚0.4395509362220764 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4122872054576874 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_1_of_4‚0.910903811454773 *jet/accuracy_3_of_6‚0.1538461595773697 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.988496720790863 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9957482814788818 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4236849546432495 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.515860915184021 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19770114123821259 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994879961013794 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 jet/accuracy_4_of_7‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15650269389152527 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.43656131625175476 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47873398661613464 6"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5180624127388 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.982475996017456 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 particle/accuracy_7_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5050670504570007 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9685291647911072 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49562782049179077 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5921541452407837 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.181142196059227 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.037540942430496216 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5555555820465088 particle/accuracy_2_of_3‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1835632473230362 'particle/f_score‚0.3966263234615326 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5699884295463562 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.48722925782203674 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 epoch‚4 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1877370923757553 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2659422755241394 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5050670504570007 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9901541471481323 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06024220213294029 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 8 Purity/2FRt1SR&Vdë qqt*FBt/FRt_purity‚0.21630118787288666 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26088669896125793 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03892729431390762 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4439571797847748 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5105076432228088 *jet/accuracy_2_of_3‚0.2529905438423157 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02533692866563797 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41447368264198303 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5001419186592102 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20053166151046753 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_4‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1754467785358429 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18817918002605438 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,jet/accuracy_3_of_3‚0.027425043284893036 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41649138927459717 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15101826190948486 particle/accuracy_2_of_4‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14925578236579895 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32351189851760864 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13973921537399292 jet/accuracy_1_of_7‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9969051480293274 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.988006591796875 ,jet/accuracy_4_of_4‚0.009997246786952019îÈ962î- _wandbruntime‚5144  _runtime‚5144˨ÌßÈï ' lr-AdamW/pg1‚0.00022718221400806383 ' lr-AdamW/pg2‚0.00022718221400806383 trainer/global_step‚6049 % _timestamp‚1.7884427043428118e+09 _runtime‚ 5144.38746651 _step‚495Ë͉´ÍðÇ % _timestamp‚1.7884427043428118e+09 _runtime‚ 5144.38746651 _step‚495 ' lr-AdamW/pg1‚0.00022718221400806383 ' lr-AdamW/pg2‚0.00022718221400806383 trainer/global_step‚6049ïPv 2ñ- _wandbruntime‚5144  _runtime‚5144O «Ó (ò†P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113017880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687539969659392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100227878 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚386 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112997846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚676667350204538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚676664726491902 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42091775 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676304736000000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593928139295744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11235260050944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89566967 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58136103 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687542596260352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42091389 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11237860260352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 % _timestamp‚1.7884427043435433e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89566961 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚35910656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11235260050944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687539969659392 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15401013353472 _step‚496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58136100 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0qËÇu(ó‹P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42091775 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676304736000000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593928139295744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11235260050944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89566967 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58136103 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚687542596260352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42091389 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11237860260352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 % _timestamp‚1.7884427043435433e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89566961 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚35910656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11235260050944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687539969659392 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15401013353472 _step‚496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58136100 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113017880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687539969659392 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100227878 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚386 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚389 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112997846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚676667350204538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚676664726491902 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636ð³¸Sí”ô‚(:‰ ÐðåÔÈý—É gpu.0.gpu‚71.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚42.0 gpu.0.powerWatts‚258.658' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.7316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿy*†4õ‚(:© ÐðåÔ‹æË$ disk./.usageGB‚1107.736515045166 disk.dm-1.out‚ 193.0703125 disk.rbd1.in‚0$ memory_percent‚77.13088989257812" proc.memory.rssMB‚ 2725.9140625 cpu‚25.335646337898197 network.recv‚ 1.888782e+06 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1754.2109375 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.rbd0.out‚0" disk.dm-0.out‚48146.3544921875 disk.drbd1777.in‚0 disk.rbd1.out‚2.453125* proc.memory.percent‚16.637659072875977 network.sent‚ 1.455168e+07 disk.dm-0.in‚ 1754.2109375 proc.cpu.threads‚17) disk./.usagePercent‚74.34112697752508 disk.sda1.out‚0 disk.sda2.in‚0$ proc.memory.availableMB‚3746.875 disk.sda1.in‚0 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.in‚0.5625' disk.nvme0c0n1.out‚48146.3544921875 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚48146.3544921875 disk.sda3.out‚ 193.0703125 disk.dm-1.in‚0.5625! disk.rbd2.out‚460.92724609375 disk.sda.out‚ 193.0703125 disk.rbd3.in‚0{¼2ö- _wandbruntime‚5145  _runtime‚5145ÑbŒÕ¸÷N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚687652368307712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202597277 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884427050774684e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚354 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89581203 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594023176872960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665554451202048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚35897344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58145299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42098198 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5145.122781135 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687649741693440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11237092379648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89581197 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113036140 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100243494 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15403562934784 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100243851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58145296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚578619578040832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚578619549906432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11218445806794 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚64031744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚687652368307712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676412649313792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89581203 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚687652368307712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202597277 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884427050774684e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚354 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89581203 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594023176872960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665554451202048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚35897344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58145299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42098198 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚5145.122781135 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚687649741693440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11237092379648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89581197 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113036140 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100243494 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15403562934784 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100243851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58145296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚578619578040832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚578619549906432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11218445806794 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚64031744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚687652368307712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676412649313792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89581203 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 Fý‚(:³ ßðåÔæÖöË disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚49013.1513671875 disk.dm-1.out‚ 193.77734375 network.recv‚ 1.895353e+06) disk./.usagePercent‚74.34106604773193 disk.sda3.in‚0.5625 proc.cpu.threads‚17! disk.rbd2.out‚461.05615234375 disk.sda.out‚ 193.77734375 disk.sda3.out‚ 193.77734375 network.sent‚ 1.4687091e+07 disk.sda1.out‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚49013.1513671875 disk.dm-1.in‚0.5625" proc.memory.rssMB‚ 2725.9140625 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚49013.1513671875$ memory_percent‚77.13096141815186 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.dm-0.in‚ 1754.2109375 disk.rbd1.out‚2.453125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1754.2109375% disk./.usageGB‚1107.7356071472168 disk.rbd0.in‚0) proc.memory.availableMB‚ 3746.86328125* proc.memory.percent‚16.637659072875977 cpu‚25.334764103164776 disk.rbd1.in‚0‹}%’þ‚(:‡ îðåÔòܸÉ gpu.0.gpu‚90.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚254.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«àÏu@ÿ‚(:µ îðåÔÌ( disk.nvme0c0n1.out‚49013.83251953125 cpu‚25.33393831584595 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625* proc.memory.percent‚16.637659072875977 network.sent‚ 1.4784669e+07 disk.sda.out‚ 194.08984375 disk.drbd1777.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.34106092758125 disk.sda.in‚0.5625 disk.dm-1.out‚ 194.08984375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 194.08984375" disk.nvme0c0n1.in‚ 1754.2109375 disk.dm-0.in‚ 1754.2109375# disk.dm-0.out‚49013.83251953125$ memory_percent‚77.97219753265381" proc.memory.rssMB‚ 2725.9140625 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚49013.83251953125) proc.memory.availableMB‚ 3609.03515625 network.recv‚ 1.900718e+06 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚2.453125! disk.rbd2.out‚462.63037109375% disk./.usageGB‚1107.7355308532715 disk.rbd3.in‚0ŸÝ¡>2€- _wandbruntime‚5183  _runtime‚5183‹c>ÇÁ & lr-AdamW/pg1‚0.0002244236582721484 & lr-AdamW/pg2‚0.0002244236582721484 trainer/global_step‚6099 % _timestamp‚1.7884427437101436e+09 _runtime‚5183.754836555 _step‚499›t8,Ì‚Æ & lr-AdamW/pg1‚0.0002244236582721484 & lr-AdamW/pg2‚0.0002244236582721484 trainer/global_step‚6099 % _timestamp‚1.7884427437101436e+09 _runtime‚5183.754836555 _step‚499óëª8t2ƒ- _wandbruntime‚5183  _runtime‚5183 {‡ („†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42428185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681701694875648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90278767 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚693028585775616 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101024540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90278767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15528235202048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598674827864576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670765194169344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11329491109376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11308097188734 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693028585775616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583146564528128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚682073265930110 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 % _timestamp‚1.7884427437108805e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113912898 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583146592662528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113912898 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11326890899968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚693031212376576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681701694875648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90278767 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204211699 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚693028585775616 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101024540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90278767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15528235202048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598674827864576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670765194169344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11329491109376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11308097188734 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693028585775616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583146564528128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚682073265930110 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 % _timestamp‚1.7884427437108805e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113912898 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583146592662528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113912898 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11326890899968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚693031212376576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42428185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648ôG³Ô2†- _wandbruntime‚5184  _runtime‚5184ÌèÀÍ'‡N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113931132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58605099 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15530758333952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11331323451392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101040357 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598769660338176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113931132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42435258 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101039922 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583238980716032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 trainer/global_step‚6099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58605096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113951192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚693138357809664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670871535343616 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚693140984423936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚693140984423936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681809660972544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204224129 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681809660972544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670871509915648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90293003 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90293003 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11328723228672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113951192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15530707756544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11331323451392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598769739049984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚682181436447050 _step‚501 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90292997 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0«€R,'ˆ”N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚583238980716032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 trainer/global_step‚6099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58605096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113951192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚693138357809664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670871535343616 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚693140984423936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚693140984423936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681809660972544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204224129 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681809660972544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670871509915648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90293003 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90293003 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11328723228672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113951192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15530707756544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11331323451392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598769739049984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚682181436447050 _step‚501 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90292997 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113931132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58605099 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15530758333952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11331323451392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101040357 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598769660338176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113931132 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42435258 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101039922õ¢¬>o2‰- _wandbruntime‚5184  _runtime‚5184 ãê¯]Š× 3loss/FBt3/assignment_loss‚0.0012898007019733388 0loss/FBt4/assignment_loss‚0.5236755218238613 /loss/FRt1/detection_loss‚1.5366091420157597 /loss/FRt2/assignment_loss‚2.845008999109268 /loss/FRt2/detection_loss‚2.3264496347733905 -loss/FBt4/detection_loss‚2.20230333507061 _step‚502 0loss/FRt1/assignment_loss‚0.9601958864010297 1loss/SRqqt1/detection_loss‚0.9364083507680334 2loss/SRqqt4/assignment_loss‚1.0837369325857322 /loss/FRt3/detection_loss‚2.4036871420987294 epoch‚5 % _timestamp‚1.7884427444494717e+09 0loss/FRt3/assignment_loss‚3.6386482384469776 /loss/FRt4/assignment_loss‚2.318760136240407 1loss/FBt1/assignment_loss‚0.07961018778073535 /loss/FRt4/detection_loss‚2.3064105514633026 /loss/FBt2/detection_loss‚1.9818907499313354 1loss/SRqqt2/detection_loss‚2.0775428609479043 0loss/SRqqt3/detection_loss‚2.184009499847889 /loss/FBt1/detection_loss‚0.6064940587834413 2loss/SRqqt1/assignment_loss‚0.7808848128879617 1loss/SRqqt2/assignment_loss‚1.625473623407916 2loss/SRqqt3/assignment_loss‚1.6385123560205102 1loss/FBt2/assignment_loss‚0.29883251873980043 trainer/global_step‚6099 1loss/SRqqt4/detection_loss‚1.6788633952920253 .loss/FBt3/detection_loss‚3.646371304988861 %loss/total_loss‚39.68166904012745 _runtime‚5184.494032693·.Wb‹Ü 3loss/FBt3/assignment_loss‚0.0012898007019733388 0loss/FBt4/assignment_loss‚0.5236755218238613 /loss/FRt1/detection_loss‚1.5366091420157597 /loss/FRt2/assignment_loss‚2.845008999109268 /loss/FRt2/detection_loss‚2.3264496347733905 -loss/FBt4/detection_loss‚2.20230333507061 _step‚502 0loss/FRt1/assignment_loss‚0.9601958864010297 1loss/SRqqt1/detection_loss‚0.9364083507680334 2loss/SRqqt4/assignment_loss‚1.0837369325857322 /loss/FRt3/detection_loss‚2.4036871420987294 epoch‚5 % _timestamp‚1.7884427444494717e+09 0loss/FRt3/assignment_loss‚3.6386482384469776 /loss/FRt4/assignment_loss‚2.318760136240407 1loss/FBt1/assignment_loss‚0.07961018778073535 /loss/FRt4/detection_loss‚2.3064105514633026 /loss/FBt2/detection_loss‚1.9818907499313354 1loss/SRqqt2/detection_loss‚2.0775428609479043 0loss/SRqqt3/detection_loss‚2.184009499847889 /loss/FBt1/detection_loss‚0.6064940587834413 2loss/SRqqt1/assignment_loss‚0.7808848128879617 1loss/SRqqt2/assignment_loss‚1.625473623407916 2loss/SRqqt3/assignment_loss‚1.6385123560205102 1loss/FBt2/assignment_loss‚0.29883251873980043 trainer/global_step‚6099 1loss/SRqqt4/detection_loss‚1.6788633952920253 .loss/FBt3/detection_loss‚3.646371304988861 %loss/total_loss‚39.68166904012745 _runtime‚5184.494032693öRKh¥Œ‚(:’ ýðåÔÖ봃 gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.36' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.47200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fŠ´A‚(:¶ ýðåÔÌ­Ë disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.637659072875977 proc.cpu.threads‚17 network.recv‚ 1.904878e+06% disk./.usageGB‚1107.7358665466309) proc.memory.availableMB‚ 1767.01171875" disk.nvme0c0n1.in‚ 1754.2109375 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda1.out‚0( disk.nvme0c0n1.out‚49014.17626953125# disk.dm-0.out‚49014.17626953125 disk.rbd1.out‚2.453125 disk.sda.out‚ 194.56640625 cpu‚25.333132259681932 disk.dm-0.in‚ 1754.2109375 disk.sda1.in‚0 disk.nvme0n1.in‚ 1754.2109375 network.sent‚ 1.4790605e+07 disk.sda3.in‚0.5625 disk.dm-1.out‚ 194.56640625 disk.sda2.in‚0& disk.nvme0n1.out‚49014.17626953125$ memory_percent‚89.21501636505127" proc.memory.rssMB‚ 2725.9140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.5625! disk.rbd2.out‚462.87255859375 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.34108345624426 disk.sda3.out‚ 194.56640625 disk.drbd1777.in‚0f‚ˆŸŽ‚(:” ŒñåÔ‡š‹É gpu.0.gpu‚81.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚270.027' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.005399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x.‡]_‚(:µ ŒñåÔŽœíÊ disk.rbd3.out‚0 cpu‚25.332325571634044 disk.sda2.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚49111.86181640625 disk.sda3.out‚ 195.03515625 disk.rbd0.out‚0 disk.sda.in‚0.5625# memory_percent‚89.2164945602417% disk./.usageGB‚1107.7358665466309 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 195.03515625 disk.rbd3.in‚0 network.sent‚ 1.4899572e+07 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.2109375# disk.dm-0.out‚49111.86181640625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0.5625" disk.nvme0c0n1.in‚ 1754.2109375 disk.rbd1.in‚0 disk.sda.out‚ 195.03515625 disk.sda1.in‚0) proc.memory.availableMB‚ 1766.76953125 network.recv‚ 1.911295e+06 disk.rbd1.out‚2.453125" proc.memory.rssMB‚ 2725.9140625 diîÛásk.drbd1777.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.637659072875977 disk.sda2.out‚0) disk./.usagePercent‚74.34108345624426( disk.nvme0c0n1.out‚49111.86181640625! disk.rbd2.out‚464.38427734375œƒD ”‚(:‰ ›ñåÔºå É gpu.0.gpu‚77.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚225.464' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.0928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îô~²9‘‚(:® ›ñåÔŠöÕÊ) proc.memory.availableMB‚ 1766.73046875 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.out‚2.453125 disk.sda.in‚0.5625 disk.sda2.out‚0# memory_percent‚89.2167329788208) disk./.usagePercent‚74.34129568649014 disk.drbd1777.in‚0 disk.sda.out‚ 195.55078125 disk.dm-0.in‚ 1754.2109375! disk.dm-0.out‚49115.728515625$ disk./.usageGB‚1107.739028930664 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1754.2109375 disk.sda1.out‚0 disk.sda3.in‚0.5625" proc.memory.rssMB‚ 2725.9140625* proc.memory.percent‚16.637659072875977 network.sent‚ 1.4905863e+07 disk.rbd2.in‚0$ disk.nvme0n1.out‚49115.728515625& disk.nvme0c0n1.out‚49115.728515625 cpu‚25.331472607555664 proc.cpu.threads‚17 network.recv‚ 1.915832e+06 disk.rbd3.out‚0 disk.sda3.out‚ 195.55078125 disk.dm-1.out‚ 195.55078125! disk.rbd2.out‚466.06396484375 disk.sda1.in‚0Í¡Xq2’- _wandbruntime‚5222  _runtime‚5222w¶øÉ“Ã % _timestamp‚1.7884427828064334e+09 _runtime‚5222.851053297 _step‚503 ' lr-AdamW/pg1‚0.00022165871166041355 ' lr-AdamW/pg2‚0.00022165871166041355 trainer/global_step‚6149åw‚kÎ”È _runtime‚5222.851053297 _step‚503 ' lr-AdamW/pg1‚0.00022165871166041355 ' lr-AdamW/pg2‚0.00022165871166041355 trainer/global_step‚6149 % _timestamp‚1.7884427828064334e+09÷6`2k2•- _wandbruntime‚5222  _runtime‚5222Öv"“ (–†P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11421121958400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42763626 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42763165 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚676082252882944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11402176198010 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚687098680142848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59055700 % _timestamp‚1.7884427828071642e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114847984 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚687098706534400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚687481805368318 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101819329 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚603419758151680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90990567 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚5222.851977087 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698517201891840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11399577913342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15653841539072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90990567 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698519828492800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698517201891840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587765995337728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101818865 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚587765967203328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676082227454976 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚687098680142848 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚603419836876800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59055703 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114847984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚676082227454976 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚687098680142848 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚603419836876800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59055703 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114847984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698517201891840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11399577913342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15653841539072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90990567 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698519828492800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698517201891840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587765995337728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101818865 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6149ødŸ2˜- _wandbruntime‚5223  _runtime‚5223(^»&™N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676188568629248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91004797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 _step‚505 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚687206619848192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101835177 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59064899 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114866244 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11404005540886 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698629600540160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587858355256832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91004803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884427835416367e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205850981 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5223.586664201 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚687592599598102 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91004803 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚676188594057216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698626973925888 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101834742 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚687206646239744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587858383391232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚687206646239744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚687589975885258 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11420354077696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91004797 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114846184 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15656311416320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698629600540160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114846184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚603514666673152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42769846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11420354077696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚603514745384960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.5mN•[current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574êÕ »'š•N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11422954300416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15656361993728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114866244 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11404005540886 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚698629600540160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587858355256832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91004803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884427835416367e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205850981 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5223.586664201 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚687592599598102 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91004803 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚676188594057216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698626973925888 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101834742 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚687206646239744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587858383391232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚687206646239744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚687589975885258 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11420354077696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91004797 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114846184 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15656311416320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698629600540160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114846184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚603514666673152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42769846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11420354077696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚603514745384960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚676188568629248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91004797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 _step‚505 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚687206619848192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101835177 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59064899 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704005818008064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207465403 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704008444609024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚692495665410048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚592385398012928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681399311596544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692892968519162 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚471 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59515503 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11510152598016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115763036 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115743002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43098644 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚608164636416512 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚692495691801600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚692495665410048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 _step‚508 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91702367 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚692495691801600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91702361 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115743002 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 trainer/global_step‚6199 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15779238403584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11510152598016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59515500 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚592385369878528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115763036 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91702361 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704008444609024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11493656922618 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102614144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43099112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15779187812864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 % _timestamp‚1.7884428216478534e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11491058637950 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704005818008064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681399286168576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚608164557691392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91702367.¿û‰§‹P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11510152598016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59515500 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚592385369878528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115763036 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91702361 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚704008444609024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11493656922618 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102614144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43099112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15779187812864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 % _timestamp‚1.7884428216478534e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11491058637950 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704005818008064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681399286168576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚608164557691392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91702367 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692890344806526 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11512752807424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102614615 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704005818008064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207465403 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/ac–9ÿƈtive_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚704008444609024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚692495665410048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚592385398012928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681399311596544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692892968519162 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚471 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59515503 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11510152598016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115763036 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115743002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43098644 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚608164636416512 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚692495691801600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚692495665410048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 _step‚508 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91702367 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚692495691801600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91702361 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115743002 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 trainer/global_step‚6199 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15779238403584üÍ7Z2¨- _wandbruntime‚5262  _runtime‚5262Åç/ñ'©N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91716603 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚608259457942016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11511984926720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102630009 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115761236 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704115590042112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59524699 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681505652770816 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91716603 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚693001139036310 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚704118216656384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115761236 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚592477786066432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91716597 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102630456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15781700009984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚592477757932032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681505627342848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚692603605115392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91716597 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚692603631506944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11514585149440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 _step‚509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11511984926720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 $ _timestamp‚1.788442822382025e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15781750587392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 _runtime‚5262.427399526 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43105313 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚692603631506944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11514585149440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43105757 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11495486265494 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115781296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚608259536653824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6199 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚704118216656384 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6199 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚704118216656384 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚608259457942016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11511984926720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102630009 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115761236 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704115590042112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59524699 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚681505652770816 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91716603 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚693001139036310 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚704118216656384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115761236 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚592477786066432 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91716597 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102630456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15781700009984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚592477757932032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚681505627342848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚692603605115392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91716597 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚692603631506944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11514585149440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 _step‚509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11511984926720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 $ _timestamp‚1.788442822382025e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15781750587392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 _runtime‚5262.427399526 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43105313 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚692603631506944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11514585149440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43105757 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11495486265494 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115781296ý¹@Ã[2«- _wandbruntime‚5262  _runtime‚5262²(u\¬Ö _runtime‚ 5262.42847895 _step‚510 2loss/SRqqt2/assignment_loss‚1.5519950260556694 /loss/FBt3/detection_loss‚2.6878405928611757 %loss/total_loss‚41.77389969634538 .loss/FRt4/detection_loss‚2.323140868118831 0loss/FRt1/assignment_loss‚1.7906492153803508 0loss/FRt2/assignment_loss‚2.7846409361413182 1loss/SRqqt1/detection_loss‚0.8864399900331217 /loss/FRt3/assignment_loss‚3.700735221890842 1loss/FBt3/assignment_loss‚0.17913056355901064 1loss/SRqqt2/detection_loss‚1.4391973886800848 1loss/SRqqt3/detection_loss‚2.4125432908535003 /loss/FBt1/detection_loss‚0.5934408887367074 epoch‚5 trainer/global_step‚6199 1loss/SRqqt4/detection_loss‚1.7406166438329018 /loss/FBt4/detection_loss‚2.9105225652456284 1loss/FBt2/assignment_loss‚0.03141223708298639 -loss/FRt3/detection_loss‚2.45444343078487 0loss/FRt4/assignment_loss‚2.7288103586151484 1loss/FBt1/assignment_loss‚0.12574341009168014 1loss/FBt4/assignment_loss‚0.12791313766501844 % _timestamp‚1.7884428223839097e+09 2loss/SRqqt1/assignment_loss‚1.0783353073467665 2loss/SRqqt3/assignment_loss‚1.4597407958337238 2loss/SRqqt4/assignment_loss‚1.5359958146335715 /loss/FRt1/detection_loss‚1.7731812223792076 .loss/FRt2/detection_loss‚2.230823539355968 .loss/FBt2/detection_loss‚3.226607251167297ÇLsta­Û 1loss/FBt2/assignment_loss‚0.03141223708298639 -loss/FRt3/detection_loss‚2.45444343078487 0loss/FRt4/assignment_loss‚2.7288103586151484 1loss/FBt1/assignment_loss‚0.12574341009168014 1loss/FBt4/assignment_loss‚0.12791313766501844 % _timestamp‚1.7884428223839097e+09 2loss/SRqqt1/assignment_loss‚1.0783353073467665 2loss/SRqqt3/assignment_loss‚1.4597407958337238 2loss/SRqqt4/assignment_loss‚1.5359958146335715 /loss/FRt1/detection_loss‚1.7731812223792076 .loss/FRt2/detection_loss‚2.230823539355968 .loss/FBt2/detection_loss‚3.226607251167297 _runtime‚ 5262.42847895 _step‚510 2loss/SRqqt2/assignment_loss‚1.5519950260556694 /loss/FBt3/detection_loss‚2.6878405928611757 %loss/total_loss‚41.77389969634538 .loss/FRt4/detection_loss‚2.323140868118831 0loss/FRt1/assignment_loss‚1.7906492153803508 0loss/FRt2/assignment_loss‚2.7846409361413182 1loss/SRqqt1/detection_loss‚0.8864399900331217 /loss/FRt3/assignment_loss‚3.700735221890842 1loss/FBt3/assignment_loss‚0.17913056355901064 1loss/SRqqt2/detection_loss‚1.4391973886800848 1loss/SRqqt3/detection_loss‚2.4125432908535003 /loss/FBt1/detection_loss‚0.5934408887367074 epoch‚5 trainer/global_step‚6199 1loss/SRqqt4/detection_loss‚1.7406166438329018 /loss/FBt4/detection_loss‚2.9105225652456284þµ/5õŸ®‚(:” ÈñåÔĹ‹É gpu.0.gpu‚59.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚269.713' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.942600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³3\<¯‚(:± ÈñåÔÀé’Ë" disk.nvme0c0n1.in‚ 1754.2109375 disk.dm-0.in‚ 1754.2109375# disk.dm-0.out‚49214.85400390625 network.recv‚ 1.930697e+06( disk.nvme0c0n1.out‚49214.85400390625 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.out‚2.453125* proc.memory.percent‚16.637659072875977$ disk./.usageGB‚1108.241111755371 disk.sda1.in‚0 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 network.sent‚ 1.5026187e+07 disk.sda2.in‚0& disk.nvme0n1.out‚49214.85400390625 disk.dm-1.in‚0.5625 disk.rbd2.out‚ 467.748046875$ memory_percent‚89.21709060668945 disk.dm-1.out‚ 196.96484375 disk.rbd1.in‚0' proc.memory.availableMB‚ 1766.671875 proc.cpu.threads‚17) disk./.usagePercent‚74.37499088613178 disk.rbd3.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2725.9140625 cpu‚25.329027089446562 disk.sda.out‚ 196.96484375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 196.96484375d=—ž°‚(:“ ×ñåÔ£Â‰É gpu.0.gpu‚59.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚98.647' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.729400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Ç&&0±‚(:¥ ×ñåÔÛüÚÊ disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1754.2109375( disk.nvme0c0n1.out‚49312.37353515625 disk.rbd2.out‚ 469.427734375# memory_percent‚89.2186164855957' proc.memory.availableMB‚ 1766.421875 network.sent‚ 1.5135816e+07 disk.rbd0.in‚0 disk.dm-1.out‚197.5625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.dm-1.in‚0.5625 disk.rbd2.in‚0" proc.memory.rssMB‚ 2725.9140625 cpu‚25.328195654324176 proc.cpu.threads‚17 network.recv‚ 1.937326e+06 disk.sda.in‚0.5625 disk.sda.out‚197.5625 disk.sda3.out‚197.5625# disk.dm-0.out‚49312.37353515625 disk.drbd1777.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚74.37485955426672 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1754.2109375 disk.sda2.in‚0* proc.memory.percent‚16.637659072875977 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚2.453125% disk./.usageGB‚1108.2391548156738& disk.nvme0n1.out‚49312.37353515625”Ó£ž²‚(:“ æñåÔ¨¡ÙÈ gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.071' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹"ö0³‚(:¥ æñåÔµ¹ÖÌ disk.rbd1.out‚2.453125 disk.sda.in‚0.5625* proc.memory.percent‚16.637659072875977% disk./.usageGB‚1108.2390975952148 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2725.9140625 network.recv‚ 1.941503e+06 disk.sda2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚49313.40771484375 disk.rbd2.out‚ 469.427734375$ memory_percent‚89.21709060668945' proc.memory.availableMB‚ 1766.671875 disk.sda2.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.in‚0 disk.rbd0.out‚0( disk./.usagePercent‚74.3748557141537& disk.nvme0n1.out‚49313.40771484375 disk.dm-0.in‚ 1754.2109375 disk.rbd2.in‚0 cpu‚25.327392142639713 disk.rbd3.out‚0 disk.sda3.out‚198.0625 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚49313.40771484375 disk.dm-1.out‚198.0625 disk.sda.out‚198.0625 proc.cpu.threads‚17 network.sent‚ 1.5141134e+07 disk.sda3.in‚0.5625" disk.nvme0c0n1.in‚ 1754.2109375 disk.drbd1777.out‚06‹¼ê2´- _wandbruntime‚5300  _runtime‚5300n„w8ɵà _runtime‚5300.742169531 _step‚511 ' lr-AdamW/pg1‚0.00021611195186417188 ' lr-AdamW/pg2‚0.00021611195186417188 trainer/global_step‚6249 % _timestamp‚1.7884428606975389e+09L³-$Î¶È ' lr-AdamW/pg2‚0.00021611195186417188 trainer/global_step‚6249 % _timestamp‚1.7884428606975389e+09 _runtime‚5300.742169531 _step‚511 ' lr-AdamW/pg1‚0.00021611195186417188ÿBü% 2·- _wandbruntime‚5300  _runtime‚5300vj/º8¸…P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11601783447040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92414161 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59975300 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698301507957370 WJDeviceStatsMonitor.on_train_batch_start/requestu ‚Ó%ed_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116678088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116658054 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11582539362558 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697892677068800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚697892650677248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚5300.743099033 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92414161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43433439 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚612909623416832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚709494434124288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚709494434124288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚612909702141952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15904850863104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11601783447040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚709497060725248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11604383656448 trainer/global_step‚6249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686716370310144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697892650677248 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92414167 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116678088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597004772553728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209092255 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597004800688128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15904901453824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43433439 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚612909623416832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚709494434124288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚709494434124288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚612909702141952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15904850863104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11601783447040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚709497060725248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11604383656448 trainer/global_step‚6249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚686716370310144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697892650677248 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248€ Ì2º- _wandbruntime‚5301  _runtime‚5301Ô¦$'»N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92428403 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92428397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613004602337280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92428397 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116676288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698000590382592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43439533 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116696348 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103424029 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚698409678474518 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 _runtime‚5301.479483627 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116696348 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚6249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92428403 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698000616774144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59984496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚597097160607232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074ôÒÖ² ¼•N ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚698000616774144 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116676288 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11606215998464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 _step‚513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11603615775744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686822686056448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116696348 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103424029 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚698409678474518 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 _runtime‚5301.479483627 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116696348 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚6249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92428403 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698000616774144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59984496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚597097160607232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92428403 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92428397 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613004602337280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92428397 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116676288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698000590382592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43439533 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43767983 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16030374998528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11696014505472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚692033403595776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚714983050240512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93125961 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601624175228928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11693414296064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚5339.628838992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚714985676841472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703289662336000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚714983050240512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60435100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚617654501733888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104203086 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601624203363328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93125967 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚714985676841472 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884428995839834e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60435103 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703289635944448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617654578361856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117573106 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93125967 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117573106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210719107 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104202635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117593140 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚692033429023744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11674020087166 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117593140 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703289635944448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124`r<«"É‹P RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚714983050240512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93125961 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601624175228928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11693414296064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚5339.628838992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚714985676841472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703289662336000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚714983050240512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60435100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚617654501733888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104203086 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601624203363328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93125967 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚714985676841472 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884428995839834e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60435103 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703289635944448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617654578361856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117573106 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93125967 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117573106 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210719107 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104202635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117593140 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚692033429023744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11674020087166 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117593140 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703289635944448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11676618371834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93125961 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚703710047395578 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703289662336000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43767535 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43767983 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16030374998528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_s*þ'¹ftart/segment.large_pool.current‚1074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11696014505472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚692033403595776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 BË‚(:³ “òåǪ̂ƒÌ network.recv‚ 1.956878e+06) disk./.usagePercent‚74.36935744034082 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd1.out‚ 2.6337890625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚49411.82568359375$ disk./.usageGB‚1108.157169342041& disk.nvme0n1.out‚49411.82568359375 disk.dm-0.in‚ 1754.2109375 disk.drbd1777.out‚0# memory_percent‚89.2188549041748( proc.memory.availableMB‚ 1766.3828125" proc.memory.rssMB‚ 2725.9140625 cpu‚25.32502612921743 disk.rbd2.out‚ 472.783203125 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 199.70703125 disk.sda3.in‚0.5625# disk.dm-0.out‚49411.82568359375* proc.memory.percent‚16.637659072875977 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 1754.2109375 disk.dm-1.out‚ 199.70703125 network.sent‚ 1.526238e+07 disk.sda1.out‚0 disk.sda3.out‚ 199.70703125f>@2Ì- _wandbruntime‚5340  _runtime‚5340ú®tw'ÍN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93140203 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692139770198016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43774754 _step‚517 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚715095448888832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104218594 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16032828265472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60444296 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117611400 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703397575649792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715095448888832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11678447714710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884429003231027e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703397602041344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703397602041344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117591340 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11695246624768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11697846847488 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617749468162560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚692139744770048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 trainer/global_step‚6299 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117591340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5340.368264835 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93140197 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93140203 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117611400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568ÁïÂÏ'ΕN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚715095448888832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104218594 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16032828265472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60444296 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117611400 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703397575649792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715095448888832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11678447714710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884429003231027e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703397602041344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703397602041344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117591340 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11695246624768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11697846847488 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617749468162560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚692139744770048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 trainer/global_step‚6299 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117591340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5340.368264835 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93140197 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93140203 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117611400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93140203 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692139770198016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43774754 _step‚517…IŠ 2Ï- _wandbruntime‚5340  _runtime‚5340ì]þ3YÐÓ .loss/FRt3/detection_loss‚2.536656903465101 epoch‚5 trainer/global_step‚6299 .loss/FRt1/detection_loss‚1.924480207752259 /loss/FBt1/detection_loss‚0.5349672645148214 1loss/SRqqt1/detection_loss‚0.9434033702401554 $loss/total_loss‚40.2318188720089 1loss/FBt1/assignment_loss‚0.09541673604466348 1loss/FBt4/assignment_loss‚0.39247282255091703 /loss/FRt2/detection_loss‚2.3443918116390705 _runtime‚5340.369776003 0loss/FRt1/assignment_loss‚0.8303744764555068 1loss/SRqqt1/assignment_loss‚1.035519184522769 2loss/SRqqt2/assignment_loss‚1.5355619625400074 0loss/FBt2/assignment_loss‚0.8609120717039331 1loss/SRqqt2/detection_loss‚1.6804685601682374 1loss/SRqqt4/detection_loss‚1.8951171344181277 _step‚518 /loss/FRt3/assignment_loss‚3.248293932407133 1loss/SRqqt4/assignment_loss‚1.139575667257579 /loss/FRt4/detection_loss‚2.3171796215077243 1loss/SRqqt3/detection_loss‚2.5239097382873297 /loss/FBt2/detection_loss‚2.8912250280380247 /loss/FRt2/assignment_loss‚2.749596400807301 /loss/FRt4/assignment_loss‚2.524968950698773 2loss/SRqqt3/assignment_loss‚1.3537313335109502 1loss/FBt3/assignment_loss‚0.13341845946464068 .loss/FBt3/detection_loss‚2.432067026694616 /loss/FBt4/detection_loss‚2.3081102073192596 % _timestamp‚1.7884429003251684e+09!£óú^ÑØ .loss/FRt3/detection_loss‚2.536656903465101 epoch‚5 trainer/global_step‚6299 .loss/FRt1/detection_loss‚1.924480207752259 /loss/FBt1/detection_loss‚0.5349672645148214 1loss/SRqqt1/detection_loss‚0.9434033702401554 $loss/total_loss‚40.2318188720089 1loss/FBt1/assignment_loss‚0.09541673604466348 1loss/FBt4/assignment_loss‚0.39247282255091703 /loss/FRt2/detection_loss‚2.3443918116390705 _runtime‚5340.369776003 0loss/FRt1/assignment_loss‚0.8303744764555068 1loss/SRqqt1/assignment_loss‚1.035519184522769 2loss/SRqqt2/assignment_loss‚1.5355619625400074 0loss/FBt2/assignment_loss‚0.8609120717039331 1loss/SRqqt2/detection_loss‚1.6804685601682374 1loss/SRqqt4/detection_loss‚1.8951171344181277 _step‚518 /loss/FRt3/assignment_loss‚3.248293932407133 1loss/SRqqt4/assignment_loss‚1.139575667257579 /loss/FRt4/detection_loss‚2.3171796215077243 1loss/SRqqt3/detection_loss‚2.5239097382873297 /loss/FBt2/detection_loss‚2.8912250280380247 /loss/FRt2/assignment_loss‚2.749596400807301 /loss/FRt4/assignment_loss‚2.524968950698773 2loss/SRqqt3/assignment_loss‚1.3537313335109502 1loss/FBt3/assignment_loss‚0.13341845946464068 .loss/FBt3/detection_loss‚2.432067026694616 /loss/FBt4/detection_loss‚2.3081102073192596 % _timestamp‚1.7884429003251684e+09†ÝÇñžÒ‚(:“ ¢òåÔΟ›É gpu.0.gpu‚93.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚263.813' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.76259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B0zB?Ó‚(:´ ¢òåÔö»ŸË& disk.nvme0n1.out‚49509.43798828125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1754.2109375( disk.nvme0c0n1.out‚49509.43798828125 cpu‚25.324220652223495 disk.sda3.out‚ 200.0390625 disk.rbd2.in‚0# disk.dm-0.out‚49509.43798828125 disk.dm-1.in‚0.5625 network.recv‚ 1.961595e+06) disk./.usagePercent‚74.36927987005795 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.2109375 network.sent‚ 1.535841e+07 disk.rbd0.out‚0 disk.drbd1777.out‚0* proc.memory.percent‚16.637659072875977 disk.sda1.in‚0 disk.dm-1.out‚ 200.0390625% disk./.usageGB‚1108.1560134887695 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.out‚ 474.294921875 disk.sda.in‚0.5625$ memory_percent‚89.21582698822021 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.6337890625 disk.sda.out‚ 200.0390625) proc.memory.availableMB‚ 1766.87890625" proc.memory.rssMB‚ 2725.9140625 é»Ú”Ô‚(:‰ ±òåÔÀüÉ gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚226.237' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.2474 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01µºv1Õ‚(:¦ ±òåÔ©ŒË disk.sda.out‚200.625 disk.sda1.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1108.1562461853027" disk.dm-0.out‚49509.9091796875 disk.drbd1777.out‚0 disk.rbd2.out‚ 474.458984375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 1754.2109375 cpu‚25.323459987899408 disk.dm-0.in‚ 1754.2109375 disk.dm-1.in‚0.5625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625* proc.memory.percent‚16.637659072875977 disk.sda3.out‚200.625 disk.rbd1.out‚ 2.6337890625 disk.sda2.out‚0% disk.nvme0n1.out‚49509.9091796875 network.sent‚ 1.5456811e+07) disk./.usagePercent‚74.36929548651754 disk.nvme0n1.in‚ 1754.2109375" proc.memory.rssMB‚ 2725.9140625 disk.dm-1.out‚200.625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚49509.9091796875 proc.cpu.threads‚17 network.recv‚ 1.967773e+06 disk.sda.in‚0.5625 disk.rbd3.out‚0$ memory_percent‚89.21597003936768) proc.memory.availableMB‚ 1766.85546875ÂX¹2Ö- _wandbruntime‚5378  _runtime‚5378¬O¾È× ' lr-AdamW/pg1‚0.00021054655812415476 ' lr-AdamW/pg2‚0.00021054655812415476 trainer/global_step‚6349 $ _timestamp‚1.788442938674782e+09 _runtime‚5378.719424742 _step‚519k•ÍØÇ ' lr-AdamW/pg1‚0.00021054655812415476 ' lr-AdamW/pg2‚0.00021054655812415476 trainer/global_step‚6349 $ _timestamp‚1.788442938674782e+09 _runtime‚5378.719424742 _step‚519‡'ô4Ä2Ù- _wandbruntime‚5378  _runtime‚5378@ÁtÑX Ú†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚708686621211648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93837761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16156006746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93837767 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚708686621211648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118508192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11787645354496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11765500811774 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720471666356736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884429386756546e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚606243606038528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16156057337344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 _step‚520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720471666356736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚606243577904128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118488158 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11787645354496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104999225ÎM(Û‹P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11787645354496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11765500811774 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720471666356736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884429386756546e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚606243606038528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16156057337344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 _step‚520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720471666356736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚606243577904128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118488158 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11787645354496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104999225 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚708686621211648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93837761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16156006746624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93837767 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚708686621211648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118508192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ˆkfÿ 2Ü- _wandbruntime‚5379  _runtime‚5379®1Cœ'ÝN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚709226757350934 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44111347 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11769928439318 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118506392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93852003 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11789477696512 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44110891 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60904099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720584065005056 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105015446 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105014987 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚708794587308544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93851997 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622494482063872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720584065005056 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118506392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16158516106240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720581438390784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚622494560775680 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118526452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚697456828911616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93852003 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93851997 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11786877473792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11786877473792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚709224133638090 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60904096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚708794587308544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118526452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11789477696512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16158566683648 _runtime‚5379.454720337 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚708794560916992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚606335994092032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚606335965957632 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697456803483648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574Þdk¼ÐÞ•N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105015446 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105014987 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚708794587308544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93851997 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622494482063872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720584065005056 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118506392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16158516106240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720581438390784 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚622494560775680 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118526452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚697456828911616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93852003 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93851997 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11786877473792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11786877473792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚709224133638090 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60904096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚708794587308544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6349 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118526452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11789477696512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16158566683648 _runtime‚5379.454720337 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚708794560916992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚606335994092032 _KDeviceStatsMo‡ë]&Knitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚606335965957632 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚697456803483648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚709226757350934 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44111347 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11769928439318 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118506392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93852003 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11789477696512 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44110891 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60904099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720584065005056 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968‰fC2ß- _wandbruntime‚5379  _runtime‚5379C _àÙ /loss/FRt2/detection_loss‚2.5638594953786757 1loss/SRqqt2/detection_loss‚1.9296932648471061 0loss/FRt3/assignment_loss‚2.8315741007144632 2loss/SRqqt4/assignment_loss‚1.5255064300220946 .loss/FBt4/detection_loss‚2.650782114929623 trainer/global_step‚6349 _runtime‚5379.456081663 0loss/FRt2/assignment_loss‚2.8962225644361403 2loss/FBt3/assignment_loss‚0.021670427662320434 .loss/FRt1/detection_loss‚1.700321279797289 .loss/FRt4/detection_loss‚2.861535712650844 /loss/FBt3/detection_loss‚2.9416133880615236 epoch‚5 0loss/FRt1/assignment_loss‚0.9001833465364244 /loss/FRt4/assignment_loss‚3.081843067918505 1loss/SRqqt3/assignment_loss‚1.569074791905127 1loss/FBt4/assignment_loss‚0.11677135946714164 /loss/FRt3/detection_loss‚2.7580746277579307 0loss/SRqqt3/detection_loss‚2.381248885079434 1loss/SRqqt4/detection_loss‚1.9750754046051398 0loss/FBt1/detection_loss‚0.46716350367229975 &loss/total_loss‚41.054745830641416 2loss/SRqqt1/assignment_loss‚0.8446753533805835 1loss/FBt1/assignment_loss‚0.15952413416781105 1loss/SRqqt1/detection_loss‚0.9301597731162424 /loss/FBt2/detection_loss‚2.3316767712434134 1loss/SRqqt2/assignment_loss‚1.193934402408752 1loss/FBt2/assignment_loss‚0.42256163088253135 % _timestamp‚1.7884429394115028e+09 _step‚522È ÿ§dáÞ /loss/FRt3/detection_loss‚2.7580746277579307 0loss/SRqqt3/detection_loss‚2.381248885079434 1loss/SRqqt4/detection_loss‚1.9750754046051398 0loss/FBt1/detection_loss‚0.46716350367229975 &loss/total_loss‚41.054745830641416 2loss/SRqqt1/assignment_loss‚0.8446753533805835 1loss/FBt1/assignment_loss‚0.15952413416781105 1loss/SRqqt1/detection_loss‚0.9301597731162424 /loss/FBt2/detection_loss‚2.3316767712434134 1loss/SRqqt2/assignment_loss‚1.193934402408752 1loss/FBt2/assignment_loss‚0.42256163088253135 % _timestamp‚1.7884429394115028e+09 _step‚522 /loss/FRt2/detection_loss‚2.5638594953786757 1loss/SRqqt2/detection_loss‚1.9296932648471061 0loss/FRt3/assignment_loss‚2.8315741007144632 2loss/SRqqt4/assignment_loss‚1.5255064300220946 .loss/FBt4/detection_loss‚2.650782114929623 trainer/global_step‚6349 _runtime‚5379.456081663 0loss/FRt2/assignment_loss‚2.8962225644361403 2loss/FBt3/assignment_loss‚0.021670427662320434 .loss/FRt1/detection_loss‚1.700321279797289 .loss/FRt4/detection_loss‚2.861535712650844 /loss/FBt3/detection_loss‚2.9416133880615236 epoch‚5 0loss/FRt1/assignment_loss‚0.9001833465364244 /loss/FRt4/assignment_loss‚3.081843067918505 1loss/SRqqt3/assignment_loss‚1.569074791905127 1loss/FBt4/assignment_loss‚0.11677135946714164Š/IÛ“â‚(:ˆ ÀòåÔ“ò‰É gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚265.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.079 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø œå‚(:³ ÏòåÔ·è‚Í network.sent‚ 1.5572391e+07 disk.rbd1.out‚ 2.69677734375 disk.sda2.in‚0& disk.nvme0n1.out‚49608.53857421875 disk.dm-1.in‚0.5625" disk.nvme0c0n1.in‚ 1754.2109375& proc.memory.availableMB‚ 1766.90625 disk.dm-0.in‚ 1754.2109375# disk.dm-0.out‚49608.53857421875 disk.sda.out‚ 201.6171875 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.637659072875977 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 476.080078125 disk.sda.in‚0.5625 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.out‚ 201.6171875 disk.rbd0.in‚0% disk./.usageGB‚1108.1561737060547 disk.dm-1.out‚ 201.6171875 disk.rbd3.in‚0( disk.nvme0c0n1.out‚49608.53857421875 cpu‚25.321929606400605) disk./.usagePercent‚74.36929062237438 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625$ memory_percent‚89.21566009521484 network.recv‚ 1.979225e+06 disk.sda1.out‚0" proc.memory.rssMB‚ 2725.9140625ÂbZŸ”æ‚(:‰ ÞòåÔ– ÈÈ gpu.0.gpu‚89.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.104' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.8208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÌÔ?ç‚(:´ ÞòåÔæØÓË% disk./.usageGB‚1108.6581916809082 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1754.2109375 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda3.out‚ 202.2265625 network.sent‚ 1.5577645e+07 network.recv‚ 1.982987e+06# disk.dm-0.out‚49609.21728515625 disk.dm-1.out‚ 202.2265625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1754.2109375( disk.nvme0c0n1.out‚49609.21728515625" proc.memory.rssMB‚ 2725.9140625 disk.dm-0.in‚ 1754.2109375 disk.rbd1.in‚0 disk.rbd2.out‚ 477.591796875 disk.rbd3.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1766.87109375* proc.memory.percent‚16.637659072875977 disk.sda2.in‚0& disk.nvme0n1.out‚49609.21728515625 cpu‚25.321101596796172 proc.cpu.threads‚17) disk./.usagePercent‚74.40298146988795 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.in‚0.5625$ memory_percent‚89.21587467193604 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda.out‚ 202.2265625Ç^ 2è- _wandbruntime‚5417  _runtime‚5417žŒÄÉéà % _timestamp‚1.7884429775268397e+09 _runtime‚5417.571502586 _step‚523 ' lr-AdamW/pg1‚0.00020775832272326857 ' lr-AdamW/pg2‚0.00020775832272326857 trainer/global_step‚6399,ÄŽ…ÎêÈ % _timestamp‚1.7884429775268397e+09 _runtime‚5417.571502586 _step‚523 ' lr-AdamW/pg1‚0.00020775832272326857 ' lr-AdamW/pg2‚0.00020775832272326857 trainer/global_step‚6399‹³)•_2ë- _wandbruntime‚5417  _runtime‚5417‰“ (ì†P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚627144545117696 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94549567 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16281536403968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610863008713728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 trainer/global_step‚6399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119423244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884429775276213e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105793410 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11856981536382 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11876675994112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119423244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚725960282472960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44438710 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714527126271994 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94549567 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11879276203520 _step‚524 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725960282472960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725962909073920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚714083632870400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94549561 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94549561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61354703N‘d©Ï'í‹P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11879276203520 _step‚524 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725960282472960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725962909073920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚714083632870400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94549561 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94549561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61354703 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚627144545117696 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94549567 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16281536403968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610863008713728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 trainer/global_step‚6399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119423244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884429775276213e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105793410 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11856981536382 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11876675994112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119423244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚725960282472960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44438710 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714527126271994 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonito© nÓBr.on_train_batch_start/active.large_pool.allocated‚94549567ŒŸÝl2î- _wandbruntime‚5418  _runtime‚5418ÂH“'ïN =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105809871 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119441504 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11858810879050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚726072681121280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119421444 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610955396767232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714635296789142 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5418.310159154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11861409163926 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚714191572575744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11878508322816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚627239450579456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94563803 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119421444 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚714191546184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61363899 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94563803 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16284053812224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44445518 $ _timestamp‚1.788442978265054e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚714191572575744 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 trainer/global_step‚6399 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105809414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702773862197248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94563797 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚726070054507008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213985241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61363896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610955368632832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16284003234816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702773887625216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11881108545536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119441504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44445972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 _step‚525 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11881108545536 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚726072681121280Ö²®¶'ð”N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119421444 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610955396767232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714635296789142 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5418.310159154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11861409163926 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚714191572575744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11878508322816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚627239450579456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94563803 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119421444 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚714191546184192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61363899 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94563803 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16284053812224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44445518 $ _timestamp‚1.788442978265054e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚714191572575744 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 trainer/global_step‚6399 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105809414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702773862197248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94563797 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚726070054507008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213985241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61363896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610955368632832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16284003234816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702773887625216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11881108545536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119441504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44445972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 _step‚525 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11881108545536 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚726072681121280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105809871 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119441504 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11858810879050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚726072681121280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307­,O2ñ- _wandbruntime‚5418  _runtime‚5418aíI[òÕ 2loss/SRqqt1/assignment_loss‚0.9064979929420532 1loss/FBt2/assignment_loss‚0.13318385613496908 /loss/FRt4/detection_loss‚2.4921166500667247 1loss/SRqqt4/detection_loss‚1.7168189192227288 -loss/FBt4/detection_loss‚2.33165727853775 $loss/total_loss‚39.3801842656716 1loss/SRqqt4/assignment_loss‚1.538088801408223 0loss/FBt4/assignment_loss‚0.2013333110313397 /loss/FRt2/detection_loss‚2.5654138084067855 .loss/FRt3/detection_loss‚2.388215628444925 1loss/SRqqt3/detection_loss‚2.4085144823597324 /loss/FBt3/detection_loss‚2.4008052945137024 _runtime‚5418.311686202 0loss/FRt1/assignment_loss‚1.2737745771805444 0loss/FRt2/assignment_loss‚2.1269750858462135 0loss/FRt4/assignment_loss‚2.5574973138367256 1loss/FBt3/assignment_loss‚0.17318887587316567 % _timestamp‚1.7884429782670515e+09 0loss/SRqqt1/detection_loss‚0.814322979048346 1loss/SRqqt2/detection_loss‚2.0661654555453706 /loss/FBt2/detection_loss‚2.7377267943488226 epoch‚5 trainer/global_step‚6399 0loss/FRt3/assignment_loss‚3.1484971955418586 2loss/SRqqt3/assignment_loss‚1.9624683943486982 1loss/FBt1/assignment_loss‚0.17397930099662662 .loss/FBt1/detection_loss‚0.557882406619481 _step‚526 2loss/SRqqt2/assignment_loss‚1.3940545900796468 /loss/FRt1/detection_loss‚1.31100527333716551)~`óÚ $loss/total_loss‚39.3801842656716 1loss/SRqqt4/assignment_loss‚1.538088801408223 0loss/FBt4/assignment_loss‚0.2013333110313397 /loss/FRt2/detection_loss‚2.5654138084067855 .loss/FRt3/detection_loss‚2.388215628444925 1loss/SRqqt3/detection_loss‚2.4085144823597324 /loss/FBt3/detection_loss‚2.4008052945137024 _runtime‚5418.311686202 0loss/FRt1/assignment_loss‚1.2737745771805444 0loss/FRt2/assignment_loss‚2.1269750858462135 0loss/FRt4/assignment_loss‚2.5574973138367256 1loss/FBt3/assignment_loss‚0.17318887587316567 % _timestamp‚1.7884429782670515e+09 0loss/SRqqt1/detection_loss‚0.814322979048346 1loss/SRqqt2/detection_loss‚2.0661654555453706 /loss/FBt2/detection_loss‚2.7377267943488226 epoch‚5 trainer/global_step‚6399 0loss/FRt3/assignment_loss‚3.1484971955418586 2loss/SRqqt3/assignment_loss‚1.9624683943486982 1loss/FBt1/assignment_loss‚0.17397930099662662 .loss/FBt1/detection_loss‚0.557882406619481 _step‚526 2loss/SRqqt2/assignment_loss‚1.3940545900796468 /loss/FRt1/detection_loss‚1.3110052733371655 2loss/SRqqt1/assignment_loss‚0.9064979929420532 1loss/FBt2/assignment_loss‚0.13318385613496908 /loss/FRt4/detection_loss‚2.4921166500667247 1loss/SRqqt4/detection_loss‚1.7168189192227288 -loss/FBt4/detection_loss‚2.33165727853775Žk¥E¸“ô‚(:ˆ íòåÔíë¢É gpu.0.gpu‚78.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚254.525' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.905 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m3@:õ‚(:¯ íòåÔÞ´œË network.recv‚ 1.987089e+06% disk./.usageGB‚1108.6561737060547" disk.nvme0c0n1.in‚ 1754.2109375 disk.sda2.out‚0 disk.sda3.out‚ 202.890625# memory_percent‚89.2176866531372) proc.memory.availableMB‚ 1766.57421875 disk.rbd1.out‚ 2.759765625 disk.sda3.in‚0.5625 disk.rbd0.out‚0 cpu‚25.32036035105245 disk.rbd1.in‚0) disk./.usagePercent‚74.40284604190232 disk.dm-0.in‚ 1754.2109375 disk.sda.out‚ 202.890625 disk.sda1.in‚0& disk.nvme0n1.out‚49706.76416015625 network.sent‚ 1.5583533e+07 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 479.244140625 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1754.2109375# disk.dm-0.out‚49706.76416015625 disk.dm-1.out‚ 202.890625( disk.nvme0c0n1.out‚49706.76416015625" proc.memory.rssMB‚ 2725.9140625* proc.memory.percent‚16.637659072875977 proc.cpu.threads‚17µZ"õ’ö‚(:‡ üòåÔ¢±òÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚269.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Ü“Ö=÷‚(:² üòåÔœîÆÌ disk.sda1.in‚0 disk.rbd3.in‚0 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2725.9140625 proc.cpu.threads‚17 disk.sda3.out‚ 203.41015625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1754.2109375% disk.nvme0n1.out‚49707.4326171875 disk.dm-0.in‚ 1754.2109375$ memory_percent‚89.21778202056885 disk.sda.in‚0.5625 disk.sda.out‚ 203.41015625" disk.dm-0.out‚49707.4326171875 disk.dm-1.out‚ 203.41015625 disk.rbd1.in‚0* proc.memory.percent‚16.637659072875977 disk.sda1.out‚0 disk.rbd3.out‚0 network.sent‚ 1.5692638e+07 disk.sda2.out‚0 disk.sda3.in‚0.5625' disk.nvme0c0n1.out‚49707.4326171875 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 1766.55859375 cpu‚25.319606512553552 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.759765625 disk.rbd2.out‚ 479.267578125( disk./.usagePercent‚74.4028675465352$ disk./.usageGB‚1108.656494140625 disk.sda2.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1754.2109375 network.recv‚ 1.993801e+06ÕHFé2ø- _wandbruntime‚5456  _runtime‚5456¬ÛÄÉùà ' lr-AdamW/pg2‚0.00020496716947751283 trainer/global_step‚6449 % _timestamp‚1.7884430166115832e+09 _runtime‚5456.656167627 _step‚527 ' lr-AdamW/pg1‚0.00020496716947751283—I–ÎúÈ ' lr-AdamW/pg1‚0.00020496716947751283 ' lr-AdamW/pg2‚0.00020496716947751283 trainer/global_step‚6449 % _timestamp‚1.7884430166115832e+09 _runtime‚5456.656167627 _step‚527¡?ߣ2û- _wandbruntime‚5456  _runtime‚5456~¤\ïiü†P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11951060545658 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731448898589184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚719480618137600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11970907052544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120338296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚5456.657357088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106588968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16406813817856 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615482411388928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚707984579736576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11968306843136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16406864408576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61814500 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731451525190144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120318262 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚731451525190144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11968306843136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719935665710202 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44773961 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95261361 BÌ#£ested_bytes.small_pool.peak‚ 3245630568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120338296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11948462260990 _step‚528 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719933041997566 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95261367 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚719480591746048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚719480618137600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106588461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44774465 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11970907052544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631889275797504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚707984605164544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884430166124158e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61814503 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615482383254528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120318262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95261361 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731448898589184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚719480591746048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95261367 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034”ç (ý‹P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106588461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44774465 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11970907052544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631889275797504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚707984605164544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884430166124158e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61814503 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615482383254528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120318262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95261361 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731448898589184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚719480591746048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95261367 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11951060545658 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731448898589184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚719480618137600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11970907052544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120338296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚5456.657357088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106588968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16406813817856 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615482411388928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚707984579736576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11968306843136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16406864408576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61814500 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731451525190144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120318262 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚731451525190144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11968306843136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719935665710202 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44773961 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95261361 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120338296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11948462260990 _step‚528 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719933041997566 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95261367 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚719480591746048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚719480618137600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040í­n2þ- _wandbruntime‚5457  _runtime‚54571 È'ÿN EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚529 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884430173481295e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120356556 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120336496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚615574799442432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚719588557842944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚719588557842944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631984090415104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731561297237504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 trainer/global_step‚6449 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720041212514506 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95275597 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚615574771308032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120336496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106604310 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44781061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731561297237504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95275603 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720043836227350 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731558670623232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11972739394560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95275597 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609š~gÝá €•N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44780614 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚708090920910848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11970139171840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11972739394560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95275597 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚529 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884430173481295e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120356556 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120336496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚615574799442432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚719588557842944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚719588557842944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631984090415104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731561297237504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 trainer/global_step‚6449 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720041212514506 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95275597 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚615574771308032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120336496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106604310 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44781061 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731561297237504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95275603 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720043836227350 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚731558670623232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 F‡‚(:³ šóåÔÈÖ‘Ë disk.rbd0.out‚0 disk.sda.out‚ 204.43359375 disk.nvme0n1.in‚ 1754.2109375% disk.nvme0n1.out‚49805.3740234375 disk.rbd0.in‚0$ memory_percent‚89.21773433685303 proc.cpu.threads‚17% disk./.usageGB‚1108.6567687988281 disk.sda1.in‚0 disk.sda3.out‚ 204.43359375" disk.dm-0.out‚49805.3740234375 disk.dm-1.in‚0.5625* proc.memory.percent‚16.637659072875977 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚49805.3740234375" proc.memory.rssMB‚ 2725.9140625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1754.2109375) proc.memory.availableMB‚ 1766.56640625 network.recv‚ 2.004825e+06 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.40288597907767 disk.rbd2.out‚ 480.947265625 network.sent‚ 1.5807757e+07 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1754.2109375 disk.dm-1.out‚ 204.43359375 cpu‚25.31814990499361¯çy”ˆ‚(:‰ ©óåÔˆ€˜Ê gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.021' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±{ š;‰‚(:° ©óåÔ««¢Ë disk.dm-1.out‚ 204.9921875% disk./.usageGB‚1108.6563453674316 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0 cpu‚25.31741956046983 disk.rbd3.in‚0% disk.nvme0n1.out‚49805.6787109375) disk./.usagePercent‚74.40285756224138 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 204.9921875 disk.rbd0.out‚0 disk.dm-1.in‚0.5625$ memory_percent‚89.22011852264404) proc.memory.availableMB‚ 1766.17578125 disk.sda1.out‚0 disk.sda.out‚ 204.9921875 disk.nvme0n1.in‚ 1754.2109375 proc.cpu.threads‚17 network.sent‚ 1.5813587e+07 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.5625" disk.nvme0c0n1.in‚ 1754.2109375* proc.memory.percent‚16.637659072875977' disk.nvme0c0n1.out‚49805.6787109375 disk.dm-0.in‚ 1754.2109375" disk.dm-0.out‚49805.6787109375 network.recv‚ 2.008936e+06 disk.sda1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2725.9140625 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 482.537109375æÑQL2Š- _wandbruntime‚5495  _runtime‚5495z'ۆɋà trainer/global_step‚6499 % _timestamp‚1.7884430554718084e+09 _runtime‚5495.516466518 _step‚531 ' lr-AdamW/pg1‚0.00020217368005904035 ' lr-AdamW/pg2‚0.00020217368005904035fVŸ:ÎŒÈ _step‚531 ' lr-AdamW/pg1‚0.00020217368005904035 ' lr-AdamW/pg2‚0.00020217368005904035 trainer/global_step‚6499 % _timestamp‚1.7884430554718084e+09 _runtime‚5495.516466518“;V2- _wandbruntime‚5495  _runtime‚5495¯ƒCJ (Ž…P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16532472894976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45108908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚725341581435774 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚724877603404800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620101785929728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12042541270266 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚736937514705408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121253348 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95973167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16532422304256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚724877603404800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚736940141306368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62274303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚636634286959104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95973161 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636634208233984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚725344205148410 trainer/global_step‚6499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107383211 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12039942985598 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _runtime‚5495.517464379 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107382729 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12062537901568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121253348 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788443055472577e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚736940141306368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62274300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚724877577013248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121233314 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12059937692160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0¼ø¬H(ŠP A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788443055472577e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚736940141306368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62274300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚724877577013248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121233314 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12059937692160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16532472894976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217226515 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45108908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚725341581435774 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚724877603404800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620101785929728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12042541270266 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚736937514705408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121253348 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95973167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16532422304256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚724877603404800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚736940141306368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62274303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚636634286959104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95973161 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636634208233984 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚725344205148410 trainer/global_step‚6499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107383211 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12039942985598 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _runtime‚5495.517464379 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107382729 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217206475 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12062537901568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121253348 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0”’$h†2- _wandbruntime‚5496  _runtime‚5496jXeŽ ‘N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚724985516718592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12061770020864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16534934623232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚725449751952714 H:DeviceStatsMonitor.on_train_batch_end/activê:„Úˆe.all.allocated‚ 217259011 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚737049913353728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107399181 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚737049913353728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 % _timestamp‚1.7884430562101924e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737047286739456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620194173983232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12064370243584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121251548 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚724985543110144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12064370243584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121271608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121251548 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚724985543110144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12044370613142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚724985516718592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95987403 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620194202117632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95987403 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121271608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107398710 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12061770020864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648]1‚ù'’•N YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚725452375665558 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 trainer/global_step‚6499 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16534934623232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚725449751952714 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217259011 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚737049913353728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107399181 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚737049913353728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 % _timestamp‚1.7884430562101924e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737047286739456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620194173983232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12064370243584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121251548 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚724985543110144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚468 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12064370243584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121271608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121251548 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚724985543110144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12044370613142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚724985516718592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95987403 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620194202117632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95987403 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121271608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107398710 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12061770020864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚724985516718592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12061770020864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚718618722591744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62734103 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚730274562280448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624721188604928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641379142847488 trainer/global_step‚6549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12151568541184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96684967 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚718618697163776 % _timestamp‚1.7884430945588014e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122168400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122148366 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16657926108160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚730752744586618 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742428757422592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚730750120873982 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12134021994874 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚730274562280448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96684967 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pooÂsúí—l.allocated‚ 122168400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742426130821632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742428757422592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574Õ<(Ÿ‹P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218853367 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108177023 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16657875517440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚730274588672000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122148366 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12154168750592 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742426130821632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62734100 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12131423710206 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12151568541184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641379064122368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624721216739328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624721188604928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641379142847488 trainer/global_step‚6549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12151568541184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96684967 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚718618697163776 % _timestamp‚1.7884430945588014e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122168400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122148366 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16657926108160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚730752744586618 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742428757422592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚730750120873982 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12134021994874 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚730274562280448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96684967 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122168400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742426130821632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742428757422592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚718618722591744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62734103 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚730274562280448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108192375 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚5535.334195215 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚730382528377344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45449079 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16660378066944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742538529469952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218885863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742538529469952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62743299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718725038338048 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12133253052874 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122186660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122166600 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218865797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12156001092608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122166600 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12133253052874 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122186660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122166600 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218865797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12156001092608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122166600 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108192375 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚5535.334195215 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚730382528377344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45449079 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16660378066944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742538529469952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218885863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742538529469952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62743299 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚C^yü 2626614272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718725038338048™Î-á02¥- _wandbruntime‚5535  _runtime‚5535K‡ƒX^¦Ø 0loss/SRqqt4/detection_loss‚1.528103827578681 %loss/total_loss‚39.22575246574337 2loss/SRqqt3/assignment_loss‚2.0016318088769913 1loss/FBt1/assignment_loss‚0.05780957412323656 /loss/FBt2/detection_loss‚2.0496701151132584 2loss/SRqqt2/assignment_loss‚1.1136849899936854 2loss/SRqqt4/assignment_loss‚1.0289283658776964 0loss/FBt2/assignment_loss‚0.2401882539379585 /loss/FBt4/detection_loss‚3.3339345455169678 0loss/FRt1/assignment_loss‚1.1638671138456889 /loss/FRt3/assignment_loss‚3.907218372120577 1loss/SRqqt2/detection_loss‚1.6805433975211506 _step‚538 0loss/FRt2/assignment_loss‚2.1834821299864697 /loss/FBt1/detection_loss‚0.5501165000402347 _runtime‚5535.335645246 2loss/SRqqt1/assignment_loss‚1.0975630815567758 .loss/FRt2/detection_loss‚2.378823194366235 /loss/FRt3/detection_loss‚2.2619394376028246 1loss/SRqqt1/detection_loss‚1.3391847565847759 1loss/FBt4/assignment_loss‚0.18422469248374304 /loss/FRt1/detection_loss‚1.4315896413422056 0loss/SRqqt3/detection_loss‚2.535543522834778 /loss/FBt3/detection_loss‚2.2888366200707178 epoch‚5 trainer/global_step‚6549 % _timestamp‚1.7884430952907908e+09 0loss/FRt4/assignment_loss‚2.5610271043944777 1loss/FBt3/assignment_loss‚0.07777012001977048 /loss/FRt4/detection_loss‚2.2300712999544645&Häc§Ý 1loss/FBt3/assignment_loss‚0.07777012001977048 /loss/FRt4/detection_loss‚2.2300712999544645 0loss/SRqqt4/detection_loss‚1.528103827578681 %loss/total_loss‚39.22575246574337 2loss/SRqqt3/assignment_loss‚2.0016318088769913 1loss/FBt1/assignment_loss‚0.05780957412323656 /loss/FBt2/detection_loss‚2.0496701151132584 2loss/SRqqt2/assignment_loss‚1.1136849899936854 2loss/SRqqt4/assignment_loss‚1.0289283658776964 0loss/FBt2/assignment_loss‚0.2401882539379585 /loss/FBt4/detection_loss‚3.3339345455169678 0loss/FRt1/assignment_loss‚1.1638671138456889 /loss/FRt3/assignment_loss‚3.907218372120577 1loss/SRqqt2/detection_loss‚1.6805433975211506 _step‚538 0loss/FRt2/assignment_loss‚2.1834821299864697 /loss/FBt1/detection_loss‚0.5501165000402347 _runtime‚5535.335645246 2loss/SRqqt1/assignment_loss‚1.0975630815567758 .loss/FRt2/detection_loss‚2.378823194366235 /loss/FRt3/detection_loss‚2.2619394376028246 1loss/SRqqt1/detection_loss‚1.3391847565847759 1loss/FBt4/assignment_loss‚0.18422469248374304 /loss/FRt1/detection_loss‚1.4315896413422056 0loss/SRqqt3/detection_loss‚2.535543522834778 /loss/FBt3/detection_loss‚2.2888366200707178 epoch‚5 trainer/global_step‚6549 % _timestamp‚1.7884430952907908e+09 0loss/FRt4/assignment_loss‚2.5610271043944777š¤e–ž¨‚(:“ åóåÔ®öŽÉ gpu.0.gpu‚94.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚232.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.544000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å~î6©‚(:« åóåÔÊ×÷Ë disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1754.2109375' disk.nvme0c0n1.out‚50001.8681640625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 1754.2109375 disk.sda3.out‚ 207.40625" proc.memory.rssMB‚ 2725.9140625" disk.dm-0.out‚50001.8681640625 disk.dm-1.in‚0.5625 disk.dm-0.in‚ 1754.2109375 disk.rbd1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚50001.8681640625* proc.memory.percent‚16.637659072875977 disk.drbd1777.in‚0 disk.sda2.in‚0$ memory_percent‚89.22183513641357 cpu‚25.314491388661196 network.sent‚ 1.6032135e+07) disk./.usagePercent‚74.40285167406809 disk.rbd2.out‚ 485.767578125 disk.sda2.out‚0% disk./.usageGB‚1108.6562576293945 disk.drbd1777.out‚0 network.recv‚ 2.030057e+06 disk.rbd0.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.5625 disk.sda.out‚ 207.40625 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 1765.89453125 disk.rbd2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 207.40625Ž E—Ÿª‚(:” ôóåÔèÛ É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚176.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.376999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦˧5«‚(:ª ôóåÔÿ˜žË network.sent‚ 1.6130361e+07 disk.dm-1.out‚ 208.140625* proc.memory.percent‚16.637659072875977 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.rbd3.in‚0 disk.sda3.out‚ 208.140625" disk.nvme0c0n1.in‚ 1754.2109375& disk.nvme0c0n1.out‚50002.501953125 disk.rbd2.in‚0 disk.rbd2.out‚ 487.302734375 disk.sda.in‚0.5625 disk.sda.out‚ 208.140625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.2109375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1754.2109375 disk.dm-1.in‚0.5625$ memory_percent‚89.22076225280762) disk./.usagePercent‚74.40285167406809 disk.sda1.out‚0 disk.sda3.in‚0.5625( proc.memory.availableMB‚ 1766.0703125 cpu‚25.313743763176774 network.recv‚ 2.036653e+06 disk.sda2.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1108.6562576293945$ disk.nvme0n1.out‚50002.501953125 disk.sda2.in‚0! disk.dm-0.out‚50002.501953125" proc.memory.rssMB‚ 2725.9140625œ–‹2¬- _wandbruntime‚5573  _runtime‚5573Ôµ0É­Ã % _timestamp‚1.7884431334628167e+09 _runtime‚5573.507444168 _step‚539 ' lr-AdamW/pg1‚0.00019658202170551054 ' lr-AdamW/pg2‚0.00019658202170551054 trainer/global_step‚6599ŠO]Î®È ' lr-AdamW/pg2‚0.00019658202170551054 trainer/global_step‚6599 % _timestamp‚1.7884431334628167e+09 _runtime‚5573.507444168 _step‚539 ' lr-AdamW/pg1‚0.00019658202170551054›èáO2¯- _wandbruntime‚5573  _runtime‚5573‘]Œ (°†P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 _step‚540 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97396761 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚735671547547648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚735671573939200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚747914746937856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚646124221831168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63193903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12222904434814 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45778398 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123083452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97396767 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646124143106048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12245799599616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123063418 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45778876 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚736161284024826 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123083452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16783602416640 trainer/global_step‚6599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97396767 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚747917373538816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5573.508852942 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63193900 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12243199390208 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚736158660312190 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12225502719482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720Œñf(±‹P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12243199390208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108972298 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚735671573939200 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108972779 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚735671547547648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚723935755877376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚747917373538816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚747914746937856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220480219 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16783551825920 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚735671573939200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚747914746937856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚646124221831168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63193903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12222904434814 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45778398 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123083452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97396767 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646124143106048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12245799599616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123063418 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45778876 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚736161284024826 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123083452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16783602416640 trainer/global_step‚6599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97396767 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚747917373538816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5573.508852942 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63193900 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12243199390208 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚736158660312190 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12225502719482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 _step‚540 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97396761 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚735671547547648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646219039134720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724042097051648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629432979333632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16786110378496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚6599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚735779513644544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884431342005675e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚735779487252992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 _runtime‚5574.245598761 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97411003 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚735779487252992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚244 §êg01 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108988062 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123101712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12245031718912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63203099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629433007468032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123081652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123081652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646219117846528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724042122479616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12224733777482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45785407 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 _step‚541 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108988506 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97411003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123101712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63203096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12247631941632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12247631941632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚736266830829130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚735779513644544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748024518971904 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074!¡x‰'´•N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629432979333632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16786110378496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚6599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚735779513644544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220492649 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884431342005675e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚735779487252992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 _runtime‚5574.245598761 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97411003 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚735779487252992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108988062 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123101712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12245031718912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63203099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚629433007468032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123081652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123081652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646219117846528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724042122479616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12224733777482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45785407 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 _step‚541 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108988506 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97411003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123101712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63203096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12247631941632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12247631941632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚736266830829130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚735779513644544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748024518971904 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646219039134720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724042097051648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648Ü;«Ì2µ- _wandbruntime‚5574  _runtime‚5574W,AƒZ¶Ô /loss/FRt2/assignment_loss‚2.134496201409234 0loss/FRt3/assignment_loss‚3.8115082771881768 %loss/total_loss‚41.50848165555069 2loss/SRqqt2/assignment_loss‚1.4575113821774721 1loss/SRqqt1/detection_loss‚0.8599510443325226 1loss/SRqqt2/detection_loss‚2.1413569279015063 epoch‚5 0loss/FRt1/assignment_loss‚1.5352145848066912 1loss/SRqqt1/assignment_loss‚0.967711782189704 1loss/FBt2/assignment_loss‚0.17173108921269886 1loss/FBt3/assignment_loss‚0.17750605025018254 .loss/FRt3/detection_loss‚2.827469197177063 1loss/SRqqt4/detection_loss‚1.8236636945913578 trainer/global_step‚6599 2loss/SRqqt3/assignment_loss‚1.6174594257559096 .loss/FBt3/detection_loss‚2.732057750225067 _step‚542 1loss/FBt4/assignment_loss‚0.17343977193481155 ,loss/FBt2/detection_loss‚3.2789153556029 -loss/FBt4/detection_loss‚2.44713819026947 2loss/SRqqt4/assignment_loss‚1.0254689292853763 $ _timestamp‚1.788443134202608e+09 _runtime‚5574.247186812 /loss/FRt1/detection_loss‚1.6974683623437001 0loss/FRt4/assignment_loss‚2.6817675546595923 1loss/FBt1/assignment_loss‚0.13856040126207433 1loss/SRqqt3/detection_loss‚2.3128400794097357 /loss/FRt2/detection_loss‚2.4329380889733634 /loss/FRt4/detection_loss‚2.5873359457442633 0loss/FBt1/detection_loss‚0.474971568847816163r¸_·Ù 1loss/FBt3/assignment_loss‚0.17750605025018254 .loss/FRt3/detection_loss‚2.827469197177063 1loss/SRqqt4/detection_loss‚1.8236636945913578 trainer/global_step‚6599 2loss/SRqqt3/assignment_loss‚1.6174594257559096 .loss/FBt3/detection_loss‚2.732057750225067 _step‚542 1loss/FBt4/assignment_loss‚0.17343977193481155 ,loss/FBt2/detection_loss‚3.2789153556029 -loss/FBt4/detection_loss‚2.44713819026947 2loss/SRqqt4/assignment_loss‚1.0254689292853763 $ _timestamp‚1.788443134202608e+09 _runtime‚5574.247186812 /loss/FRt1/detection_loss‚1.6974683623437001 0loss/FRt4/assignment_loss‚2.6817675546595923 1loss/FBt1/assignment_loss‚0.13856040126207433 1loss/SRqqt3/detection_loss‚2.3128400794097357 /loss/FRt2/detection_loss‚2.4329380889733634 /loss/FRt4/detection_loss‚2.5873359457442633 0loss/FBt1/detection_loss‚0.47497156884781616 /loss/FRt2/assignment_loss‚2.134496201409234 0loss/FRt3/assignment_loss‚3.8115082771881768 %loss/total_loss‚41.50848165555069 2loss/SRqqt2/assignment_loss‚1.4575113821774721 1loss/SRqqt1/detection_loss‚0.8599510443325226 1loss/SRqqt2/detection_loss‚2.1413569279015063 epoch‚5 0loss/FRt1/assignment_loss‚1.5352145848066912 1loss/SRqqt1/assignment_loss‚0.967711782189704 1loss/FBt2/assignment_loss‚0.17173108921269886žÖ1p9¸‚(:® ƒôåÔÎÎÓÊ disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 208.8046875 disk.rbd2.out‚ 487.380859375 disk.rbd3.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1765.85546875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.759765625 network.sent‚ 1.6136592e+07 disk.sda3.in‚0.5625& disk.nvme0c0n1.out‚50100.033203125! disk.dm-0.out‚50100.033203125) disk./.usagePercent‚74.40287189866329 disk.sda3.out‚ 208.8046875" disk.nvme0c0n1.in‚ 1754.2109375 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2725.9140625 disk.nvme0n1.in‚ 1754.2109375 disk.rbd2.in‚0 disk.sda.out‚ 208.8046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚89.22207355499268 disk.dm-0.in‚ 1754.2109375* proc.memory.percent‚16.637659072875977$ disk.nvme0n1.out‚50100.033203125 disk.drbd1777.in‚0 disk.sda2.out‚0 cpu‚25.313043688942066 network.recv‚ 2.041138e+06% disk./.usageGB‚1108.65655899047856]g`¹‚(:’ „ôåÔ•ð gpu.0.gpu‚92.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.39' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.877999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£W÷£Ÿº‚(:” ’ôåÔžÑÉ gpu.0.gpu‚55.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚130.463' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.092599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ w0A»‚(:¶ ’ôåÔ’†žË disk.rbd3.out‚0 network.sent‚ 1.6244985e+07 network.recv‚ 2.047012e+06# disk.dm-0.out‚50100.92724609375 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1754.2265625$ memory_percent‚89.22107219696045" proc.memory.rssMB‚ 2725.9140625 cpu‚25.31233637720714 disk.dm-0.in‚ 1754.2265625 disk.dm-1.out‚ 209.21484375 disk.sda2.out‚0 disk.dm-1.in‚0.5625* proc.memory.percent‚16.637659072875977 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0) disk./.usagePercent‚74.40287113064069" disk.nvme0c0n1.in‚ 1754.2265625 disk.rbd1.in‚0 disk.rbd2.out‚ 488.919921875 disk.sda.in‚0.5625 disk.sda.out‚ 209.21484375 disk.sda1.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚50100.92724609375 disk.sda3.in‚0.5625 disk.sda3.out‚ 209.21484375 disk.rbd0.out‚0) proc.memory.availableMB‚ 1766.01953125% disk./.usageGB‚1108.6565475463867( disk.nvme0c0n1.out‚50100.92724609375 disk.rbd1.out‚ 2.75976562539cŸ¼‚(:” ¡ôåÔ®ÚÈ gpu.0.gpu‚89.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚249.241' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.848200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÀi:½‚(:¯ ¡ôåÔ©ÎÜË# disk.dm-0.out‚50101.20458984375) disk./.usagePercent‚74.40287394672356 disk.drbd1777.out‚0 disk.rbd2.out‚ 490.556640625 disk.sda2.out‚0 disk.dm-0.in‚ 1754.2265625 disk.dm-1.in‚0.5625% disk./.usageGB‚1108.6565895080566 disk.rbd0.in‚0 disk.dm-1.out‚ 209.859375 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda.out‚ 209.859375 disk.sda2.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚50101.20458984375* proc.memory.percent‚16.637659072875977 network.sent‚ 1.6250947e+07 disk.rbd0.out‚0$ memory_percent‚89.22247886657715( proc.memory.availableMB‚ 1765.7890625 disk.sda1.out‚0 network.recv‚ 2.051284e+06 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.nvme0n1.in‚ 1754.2265625" proc.memory.rssMB‚ 2725.9140625 cpu‚25.31163384112288& disk.nvme0n1.out‚50101.20458984375 disk.sda3.out‚ 209.859375" disk.nvme0c0n1.in‚ 1754.2265625¨ÌÑ2¾- _wandbruntime‚5612  _runtime‚56126`_÷È¿Â ' lr-AdamW/pg1‚0.00019378501806367304 ' lr-AdamW/pg2‚0.00019378501806367304 trainer/global_step‚6649 % _timestamp‚1.7884431725809858e+09 _runtime‚ 5612.62570282 _step‚543ëýÍÀÇ _runtime‚ 5612.62570282 _step‚543 ' lr-AdamW/pg1‚0.00019378501806367304 ' lr-AdamW/pg2‚0.00019378501806367304 trainer/global_step‚6649 % _timestamp‚1.7884431725809858e+09Ÿœ2Á- _wandbruntime‚5612  _runtime‚5612Ïn¥Ï (†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123978470 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚753405989655040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12334830239232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12314385159422 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12334830239232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16909134919680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222107071 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚753405989655040 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63653700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650869205502976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚753403363054080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _step‚544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚741068532814848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚741567199750398 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109767824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚729252840018944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚741068559206400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123998504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63653703 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚729252814590976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 trainer/global_step‚6649 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98108561 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884431725817542e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46114588 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98108567 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚741569823463034 _runtime‚5612.626645887 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574« 1׈ËP TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46114124 @:DeviceStatsMonitor.on_train_batch_start/ ÊÖ#‰'num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚741068532814848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98108561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123978470 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650869128875008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚753403363054080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚633959993955328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12337430448640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109768291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123998504 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98108567 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚633960022089728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98108561 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884431725817542e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46114588 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98108567 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚741569823463034 _runtime‚5612.626645887 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123978470 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚753405989655040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12334830239232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12314385159422 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12334830239232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16909134919680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222107071 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚753405989655040 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63653700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650869205502976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚753403363054080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _step‚544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚741068532814848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚741567199750398 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109767824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚729252840018944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚741068559206400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123998504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63653703 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚729252814590976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 trainer/global_step‚6649 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109783586 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650964110700544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚729359155765248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12318812786966 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98122803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46121127 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123996704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109784026 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16911700557312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46120690 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚753515761702400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16911649979904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12336662567936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚545 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124016764 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚634052382008832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚741176498911744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222139567 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 trainer/global_step‚6649 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12316214502090 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 trainer/global_step‚6649 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12316214502090 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109783586 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650964110700544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚729359155765248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12318812786966 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98122803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46121127 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123996704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109784026 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16911700557312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46120690 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚753515761702400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16911649979904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12336662567936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚545 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124016764 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚634052382008832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚741176498911744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648¡#øÎ2Ç- _wandbruntime‚5613  _runtime‚5613ª\ÑE\ÈÖ 0loss/FRt4/assignment_loss‚2.7548637599856765 1loss/FBt1/assignment_loss‚0.20895465987843864 1loss/FBt2/assignment_loss‚0.03919345102622174 2loss/FBt4/assignment_loss‚0.022001735167577863 /loss/FBt1/detection_loss‚0.5893074208342781 $ _timestamp‚1.788443173320658e+09 _step‚546 /loss/FRt1/detection_loss‚1.5825088627084538 /loss/FRt3/detection_loss‚2.6123442014779457 /loss/FRt4/detection_loss‚2.1882166818336204 .loss/FBt3/detection_loss‚2.407864212989807 &loss/total_loss‚41.105496404872525 /loss/FRt2/assignment_loss‚2.685506559080548 1loss/SRqqt1/detection_loss‚1.1143235260689701 /loss/FBt2/detection_loss‚2.5020895659923554 epoch‚5 2loss/SRqqt1/assignment_loss‚1.2224807319774602 1loss/SRqqt3/assignment_loss‚1.858562691557792 2loss/SRqqt4/assignment_loss‚1.5259284678389107 .loss/FBt4/detection_loss‚3.066719800233841 /loss/FRt3/assignment_loss‚3.212180834157126 2loss/SRqqt2/assignment_loss‚1.4129826497048774 1loss/SRqqt2/detection_loss‚1.8149329680711666 _runtime‚5613.365228319 0loss/FRt1/assignment_loss‚1.1706259879801009 0loss/SRqqt3/detection_loss‚2.319581993164555 trainer/global_step‚6649 0loss/FBt3/assignment_loss‚0.3550865814710657 .loss/FRt2/detection_loss‚2.721927855577734 1loss/SRqqt4/detection_loss‚1.7173112060940057¶¯3(UÉÛ /loss/FRt1/detection_loss‚1.5825088627084538 /loss/FRt3/detection_loss‚2.6123442014779457 /loss/FRt4/detection_loss‚2.1882166818336204 .loss/FBt3/detection_loss‚2.407864212989807 &loss/total_loss‚41.105496404872525 /loss/FRt2/assignment_loss‚2.685506559080548 1loss/SRqqt1/detection_loss‚1.1143235260689701 /loss/FBt2/detection_loss‚2.5020895659923554 epoch‚5 2loss/SRqqt1/assignment_loss‚1.2224807319774602 1loss/SRqqt3/assignment_loss‚1.858562691557792 2loss/SRqqt4/assignment_loss‚1.5259284678389107 .loss/FBt4/detection_loss‚3.066719800233841 /loss/FRt3/assignment_loss‚3.212180834157126 2loss/SRqqt2/assignment_loss‚1.4129826497048774 1loss/SRqqt2/detection_loss‚1.8149329680711666 _runtime‚5613.365228319 0loss/FRt1/assignment_loss‚1.1706259879801009 0loss/SRqqt3/detection_loss‚2.319581993164555 trainer/global_step‚6649 0loss/FBt3/assignment_loss‚0.3550865814710657 .loss/FRt2/detection_loss‚2.721927855577734 1loss/SRqqt4/detection_loss‚1.7173112060940057 0loss/FRt4/assignment_loss‚2.754863759UbþF 9856765 1loss/FBt1/assignment_loss‚0.20895465987843864 1loss/FBt2/assignment_loss‚0.03919345102622174 2loss/FBt4/assignment_loss‚0.022001735167577863 /loss/FBt1/detection_loss‚0.5893074208342781 $ _timestamp‚1.788443173320658e+09 _step‚546¢Ñ[‡ñ”Ê‚(:‰ °ôåÔïâŸÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚227.321' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.4642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õµ÷8Ë‚(:­ °ôåÔˆË•Ë disk.nvme0n1.in‚ 1754.2265625# memory_percent‚89.2214298248291! disk.dm-0.out‚50199.173828125 disk.rbd1.in‚0* proc.memory.percent‚16.637659072875977 disk.rbd2.out‚ 490.595703125 disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 210.23828125" disk.nvme0c0n1.in‚ 1754.2265625 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0) disk./.usagePercent‚74.40286985060301 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 210.23828125 disk.rbd0.out‚0 disk.dm-1.out‚ 210.23828125( proc.memory.availableMB‚ 1765.9609375 network.recv‚ 2.055629e+06$ disk.nvme0n1.out‚50199.173828125 disk.dm-0.in‚ 1754.2265625" proc.memory.rssMB‚ 2725.9140625 disk.sda2.out‚0 disk.rbd3.out‚0 network.sent‚ 1.625682e+07% disk./.usageGB‚1108.6565284729004 disk.sda2.in‚0 disk.drbd1777.out‚0 cpu‚25.31094451762096& disk.nvme0c0n1.out‚50199.173828125 disk.rbd1.out‚ 2.759765625ß*ѳ”Ì‚(:‰ ¿ôåÔƒý®É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚267.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó°·8Í‚(:­ ¿ôåÔ¥³Ë disk.rbd1.out‚ 2.759765625) proc.memory.availableMB‚ 1765.82421875" proc.memory.rssMB‚ 2725.9140625 cpu‚25.31020753314401% disk./.usageGB‚1108.6565284729004 disk.nvme0n1.in‚ 1754.2265625" disk.nvme0c0n1.in‚ 1754.2265625& disk.nvme0c0n1.out‚50199.466796875 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.637659072875977$ disk.nvme0n1.out‚50199.466796875 disk.sda3.in‚0.5625 disk.sda3.out‚ 210.9453125 network.sent‚ 1.6366499e+07 network.recv‚ 2.062224e+06 disk.dm-0.in‚ 1754.2265625$ memory_percent‚89.22226428985596 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-1.out‚ 210.9453125 disk.sda.out‚ 210.9453125 disk.sda1.in‚0 disk.dm-1.in‚0.5625) disk./.usagePercent‚74.40286985060301 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 492.107421875! disk.dm-0.out‚50199.466796875 disk.drbd1777.in‚01l$ì2Î- _wandbruntime‚5651  _runtime‚56517Æ£üÉÏà % _timestamp‚1.7884432117007067e+09 _runtime‚5651.745352385 _step‚547 ' lr-AdamW/pg1‚0.00019098800859271335 ' lr-AdamW/pg2‚0.00019098800859271335 trainer/global_step‚6699Û[²ÎÐÈ _step‚547 ' lr-AdamW/pg1‚0.00019098800859271335 ' lr-AdamW/pg2‚0.00019098800859271335 trainer/global_step‚6699 % _timestamp‚1.7884432117007067e+09 _runtime‚5651.745352385£©Ï2Ñ- _wandbruntime‚5651  _runtime‚5651 nãh (Ò†P UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚758891979170304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98820367 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124893522 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223713883 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98820367 trainer/global_step‚6699 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚746465518082048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚746978362901242 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚734569873304576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12429061297664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12426461088256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17034383194624 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124893522 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚758891979170304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚734569898732544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98820361 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚655613727137280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12405865884030 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98820361 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 _step‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64113503 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223733923 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64113500 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17034330506752 % _timestamp‚1.7884432117014523e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚746465544473600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12429061297664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚746465544473600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46450391 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638579396630528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚758891979170304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚734569898732544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98820361 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚655613727137280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12405865884030 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98820361 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 _step‚548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64113503 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223733923 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64113500 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17034330506752 % _timestamp‚1.7884432117014523e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚746465544473600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12429061297664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚746465544473600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46450391 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638579396630528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655708701096448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚734676214478848 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98834597 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98834597 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98834603 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110580154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655708618190336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17036833506304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746573457787392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5652.479201557 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12428293416960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746573484178944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46457004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110579700 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124931816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46457455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884432124340656e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638671784684032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚734676239906816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12410293511574 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12407695226698 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚747086533418390 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 cODevicö3cq_eStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638671812818432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚746573457787392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12430893639680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17036888278016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648CÙ'Ö•N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110579700 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124931816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46457455 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884432124340656e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638671784684032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚734676239906816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12410293511574 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12407695226698 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚747086533418390 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638671812818432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚746573457787392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12430893639680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17036888278016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655708701096448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚734676214478848 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98834597 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98834597 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98834603 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110580154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655708618190336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17036833506304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚549 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746573457787392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5652.479201557 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12428293416960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746573484178944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46457004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764380595286528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚643198827440128 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125828608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225360775 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _step‚552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111359928 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99532161 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚739886957446144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚660358511327232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚6749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17159712021504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884432505623136e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46786625 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚752384278626814 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764383221887488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764383221887488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64573300 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚739886932018176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125808574 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12518091937280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12499944893306 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64573303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660358590052352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17159762612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46786166 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12497346608638 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚751862529740800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125808574 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12518091937280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚752386902339450 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111359466 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99532167 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125828608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12520692146688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚751862503349248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574=ßÔ¸® å‹P YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚752386902339450 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111359466 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99532167 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125828608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12520692146688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚751862503349248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764380595286528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_sta¼I^³crt/inactive_split_bytes.large_pool.allocated‚643198827440128 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125828608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225360775 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _step‚552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111359928 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99532161 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚739886957446144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3300917248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚660358511327232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚6749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17159712021504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884432505623136e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46786625 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚752384278626814 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764383221887488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764383221887488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64573300 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚739886932018176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125808574 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12518091937280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12499944893306 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64573303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660358590052352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17159762612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46786166 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12497346608638 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚751862529740800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125808574 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12518091937280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¨Uýs<2æ- _wandbruntime‚5691  _runtime‚5691«ç'çŽN B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚752495072856598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64582499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764492993934848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 trainer/global_step‚6749 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125846868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17162271367680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64582496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111375384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660453408149504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12501774236182 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚751970469446144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚739993298620416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125826808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225373205 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111375825 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46792888 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚752492449143754 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764490367320576 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125846868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚643291187359232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 # _timestamp‚1.78844325130583e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12522524488704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12499175951306 _runtime‚5691.351196146 _step‚553 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125826808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764492993934848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12519924265984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46793326 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99546403 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0]˜x%'è“N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 # _timestamp‚1.78844325130583e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12522524488704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12499175951306 _runtime‚5691.351196146 _step‚553 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125826808 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764492993934848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12519924265984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46793326 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99546403 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚752495072856598 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64582499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764492993934848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 trainer/global_step‚6749 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125846868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17162271367680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64582496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111375384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660453408149504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12501774236182 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚751970469446144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚739993298620416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125826808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225373205 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111375825 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46792888 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚752492449143754 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764490367320576 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125846868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚643291187359232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0©°Qþ‡2é- _wandbruntime‚5691  _runtime‚5691PÝ€ô]ê× 1loss/SRqqt3/assignment_loss‚1.701901997479194 1loss/FBt2/assignment_loss‚0.07810433393619447 /loss/FRt1/detection_loss‚1.5482354811107626 /loss/FBt2/detection_loss‚1.9976792227138171 _step‚554 0loss/FRt3/assignment_loss‚3.1620961400595577 .loss/FRt4/detection_loss‚2.080680731187264 /loss/FBt4/detection_loss‚2.6340868920087814 epoch‚5 trainer/global_step‚6749 0loss/FRt2/assignment_loss‚2.4323473419898596 1loss/FBt4/assignment_loss‚0.27573837514501065 .loss/FRt3/detection_loss‚2.380216727165258 0loss/SRqqt2/detection_loss‚1.799354144505092 -loss/FBt1/detection_loss‚0.50051797118779 2loss/SRqqt2/assignment_loss‚1.4539395135884383 2loss/SRqqt4/assignment_loss‚1.7632814364469782 2loss/FBt3/assignment_loss‚0.005898252556410928 1loss/SRqqt1/detection_loss‚0.9279724900595477 $loss/total_loss‚39.6151625760662 0loss/FRt1/assignment_loss‚1.2170677841926107 /loss/FBt3/detection_loss‚3.3096139232317605 2loss/SRqqt1/assignment_loss‚0.9259547751519035 1loss/SRqqt4/detection_loss‚1.9588525204025968 _runtime‚5691.352327558 2loss/FBt1/assignment_loss‚0.045421977204931734 .loss/FRt2/detection_loss‚2.425620736984106 1loss/SRqqt3/detection_loss‚2.2096837162971497 0loss/FRt4/assignment_loss‚2.7808960914611816 % _timestamp‚1.7884432513077402e+09™òbëÜ 0loss/FRt3/assignment_loss‚3.1620961400595577 .loss/FRt4/detection_loss‚2.080680731187264 /loss/FBt4/detection_loss‚2.6340868920087814 epoch‚5 trainer/global_step‚6749 0loss/FRt2/assignment_loss‚2.4323473419898596 1loss/FBt4/assignment_loss‚0.27573837514501065 .loss/FRt3/detection_loss‚2.380216727165258 0loss/SRqqt2/detection_loss‚1.799354144505092 -loss/FBt1/detection_loss‚0.50051797118779 2loss/SRqqt2/assignment_loss‚1.4539395135884383 2loss/SRqqt4/assignment_loss‚1.7632814364469782 2loss/FBt3/assignment_loss‚0.005898252556410928 1loss/SRqqt1/detection_loss‚0.9279724900595477 $loss/total_loss‚39.6151625760662 0loss/FRt1/assignment_loss‚1.2170677841926107 /loss/FBt3/detection_loss‚3.3096139232317605 2loss/SRqqt1/assignment_loss‚0.9259547751519035 1loss/SRqqt4/detection_loss‚1.9588525204025968 _runtime‚5691.352327558 2loss/FBt1/assignment_loss‚0.045421977204931734 .loss/FRt2/detection_loss‚2.425620736984106 1loss/SRqqt3/detection_loss‚2.2096837162971497 0loss/FRt4/assignment_loss‚2.7808960914611816 % _timestamp‚1.7884432513077402e+09 1loss/SRqqt3/assignment_loss‚1.701901997479194 1loss/FBt2/assignment_loss‚0.07810433393619447 /loss/FRt1/detection_loss‚1.5482354811107626 /loss/FBt2/detection_loss‚1.9976792227138171 _step‚554ª@ r“Ÿì‚(:” ûôåÔý‚É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚44.0 gpu.0.powerWatts‚188.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s–3&:í‚(:¯ ûôåÔå¿ÄË disk.dm-0.in‚ 1754.2265625" disk.dm-0.out‚50395.8388671875) disk./.usagePercent‚74.40287804284411 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.2265625' disk.nvme0c0n1.out‚50395.8388671875 disk.dm-1.out‚ 213.2734375* proc.memory.percent‚16.637659072875977% disk.nvme0n1.out‚50395.8388671875 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.out‚ 495.462890625" proc.memory.rssMB‚ 2725.9140625$ disk./.usageGB‚1108.656650543213 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0$ memory_percent‚89.22302722930908 cpu‚25.307480259190143 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda.out‚ 213.2734375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1754.2265625 network.sent‚ 1.649194e+07 disk.sda3.out‚ 213.2734375 disk.rbd0.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1765.69921875 network.recv‚ 2.079806e+068?tŸî‚(:” ŠõåÔ¬½óÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚46.0 gpu.0.powerWatts‚243.544' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.708800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 õêÿˆï‚(:£ ŠõåÔבּË proc.cpu.threads‚17 disk.rbd2.out‚ 497.142578125 disk.sda1.in‚0 disk.sda3.in‚0.5625$ disk./.usageGB‚1108.656837463379 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0.5625* proc.memory.percent‚16.637659072875977 network.recv‚ 2.085442e+06) disk./.usagePercent‚74.40289058721329 disk.rbd1.out‚ 2.759765625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 1754.2265625 disk.drbd1777.in‚0 cpu‚25.306808908032522 disk.sda.out‚213.75 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚89.22238349914551 ne.%!¦twork.sent‚ 1.6600296e+07 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚50396.31689453125 disk.dm-0.in‚ 1754.2265625 disk.dm-1.out‚213.75" proc.memory.rssMB‚ 2725.9140625 disk.sda2.in‚0( proc.memory.availableMB‚ 1765.8046875 disk.nvme0n1.in‚ 1754.2265625& disk.nvme0n1.out‚50396.31689453125 disk.sda3.out‚213.75# disk.dm-0.out‚50396.31689453125éÍ 2ð- _wandbruntime‚5729  _runtime‚5729¸ùÉñà _runtime‚5729.800233167 _step‚555 ' lr-AdamW/pg1‚0.00018539630361349396 ' lr-AdamW/pg2‚0.00018539630361349396 trainer/global_step‚6799 % _timestamp‚1.7884432897555647e+09Î^¼ÎòÈ ' lr-AdamW/pg1‚0.00018539630361349396 ' lr-AdamW/pg2‚0.00018539630361349396 trainer/global_step‚6799 % _timestamp‚1.7884432897555647e+09 _runtime‚5729.800233167 _step‚555«º•C2ó- _wandbruntime‚5729  _runtime‚5729Uïõ (ôŠP R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚769869211402752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226987627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17284893422080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12609722786304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚769869211402752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12609722786304 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757259488616448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 _step‚556 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757259515008000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12612322995712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126723626 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65033103 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100243961 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚769871838003712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226967587 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647818201980928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757259515008000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100243961 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126743660 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47121406 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚665103176225280 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚665103095403008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚757259488616448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65033100 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647818230115328 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚769871838003712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12588827333246 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745203990731776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚5729.801219555 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚6799Ý=Ë(õP ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757259515008000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12612322995712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126723626 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65033103 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100243961 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚769871838003712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226967587 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647818201980928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2648 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757259515008000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100243961 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126743660 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47121406 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚665103176225280 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚665103095403008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚757259488616448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65033100 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647818230115328 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚769871838003712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12588827333246 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745203990731776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _runtime‚5729.801219555 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚6799 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚769869211402752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226987627 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17284893422080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12609722786304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚769869211402752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12609722786304 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757259488616448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 _step‚556 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624¬ÙZ}”ö‚(:‰ ™õåÔÝ‹¥É gpu.0.gpu‚75.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚279.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qþ¬=÷‚(:² ™õåÔɦ«Ë disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0$ memory_percent‚89.22204971313477) disk./.usagePercent‚74.40290722770303 disk.sda.in‚0.5625 disk.dm-0.in‚ 1754.2265625 disk.rbd2.out‚ 497.142578125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.5625" proc.memory.rssMB‚ 2725.9140625 network.sent‚ 1.6606191e+07 network.recv‚ 2.089509e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚50396.61572265625* proc.memory.percent‚16.637659072875977 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 214.2734375 disk.rbd0.in‚0 disk.dm-1.out‚ 214.2734375 disk.drbd1777.in‚0 disk.sda.out‚ 214.2734375" disk.nvme0c0n1.in‚ 1754.2265625' proc.memory.availableMB‚ 1765.859375% disk./.usageGB‚1108.6570854187012 disk.sda2.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚50396.61572265625 disk.sda3.in‚0.5625# disk.dm-0.out‚50396.61572265625 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 1754.2265625 cpu‚25.306140233924367 disk.rbd1.in‚0xGJ¯2ø- _wandbruntime‚5730  _runtime‚5730ïmÖæê%ù”N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665197979021824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12614155337728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112170455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚769981610051072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112170004 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚665198059830784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745310357334016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚757903612294806 trainer/global_step‚6799 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚769978983436800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126761920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647910590034432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12593254960790 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126761920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚647910618168832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17287441661952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12611555115008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100258203 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757367428321792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65042296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ZMDeviceStatsMonitor.on_t.Éd0rain_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100258203 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12611555115008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0'Cšf'ú™N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126761920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚647910618168832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17287441661952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12611555115008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100258203 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757367428321792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65042296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100258203 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12611555115008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665197979021824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12614155337728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112170455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚769981610051072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112170004 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚665198059830784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745310357334016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚757903612294806 trainer/global_step‚6799 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚769978983436800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126761920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647910590034432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12593254960790­ 0Óå2û- _wandbruntime‚5730  _runtime‚5730ï(zü`üÚ .loss/FRt1/detection_loss‚1.721201272600371 /loss/FRt2/detection_loss‚2.5995167841513953 /loss/FBt1/detection_loss‚0.5417007354406428 0loss/FRt3/assignment_loss‚3.4393579718228935 2loss/FBt2/assignment_loss‚0.007038674394910534 0loss/FBt3/assignment_loss‚0.5877018632433776 1loss/SRqqt1/detection_loss‚0.9865323570320161 epoch‚5 0loss/FRt2/assignment_loss‚2.7432497574223413 /loss/FRt3/detection_loss‚2.2140477482053273 1loss/FBt1/assignment_loss‚0.07253343644226641 1loss/FBt4/assignment_loss‚0.18928379245335236 %loss/total_loss‚40.81426627412243 0loss/FRt4/assignment_loss‚2.7148598494629064 1loss/SRqqt1/assignment_loss‚0.854090720383545 /loss/FRt4/detection_loss‚2.2399897097299495 1loss/SRqqt3/detection_loss‚2.6155246140495425 1loss/SRqqt2/detection_loss‚1.9139146615158429 1loss/SRqqt4/detection_loss‚1.9007119440370135 trainer/global_step‚6799 _runtime‚5730.540653058 0loss/FRt1/assignment_loss‚1.1325834250968436 2loss/SRqqt3/assignment_loss‚1.7834506996216313 2loss/SRqqt2/assignment_loss‚1.3852876015007496 .loss/FBt2/detection_loss‚3.508585572242737 /loss/FBt3/detection_loss‚2.1291346848011017 _step‚558 /loss/FBt4/detection_loss‚1.9214439168572426 % _timestamp‚1.7884432904957829e+09 2loss/SRqqt4/assignment_loss‚1.6125244816144308È)^eýß /loss/FBt1/detection_loss‚0.5417007354406428 0loss/FRt3/assignment_loss‚3.4393579718228935 2loss/FBt2/assignment_loss‚0.007038674394910534 0loss/FBt3/assignment_loss‚0.5877018632433776 1loss/SRqqt1/detection_loss‚0.9865323570320161 epoch‚5 0loss/FRt2/assignment_loss‚2.7432497574223413 /loss/FRt3/detection_loss‚2.2140477482053273 1loss/FBt1/assignment_loss‚0.07253343644226641 1loss/FBt4/assignment_loss‚0.18928379245335236 %loss/total_loss‚40.81426627412243 0loss/FRt4/assignment_loss‚2.7148598494629064 1loss/SRqqt1/assignment_loss‚0.854090720383545 /loss/FRt4/detection_loss‚2.2399897097299495 1loss/SRqqt3/detection_loss‚2.6155246140495425 1loss/SRqqt2/detection_loss‚1.9139146615158429 1loss/SRqqt4/detection_loss‚1.9007119440370135 trainer/global_step‚6799 _runtime‚5730.540653058 0loss/FRt1/assignment_loss‚1.1325834250968436 2loss/SRqqt3/assignment_loss‚1.7834506996216313 2loss/SRqqt2/assignment_loss‚1.3852876015007496 .loss/FBt2/detection_loss‚3.508585572242737 /loss/FBt3/detection_loss‚2.1291346848011017 _step‚558 /loss/FBt4/detection_loss‚1.9214439168572426 % _timestamp‚1.7884432904957829e+09 2loss/SRqqt4/assignment_loss‚1.6125244816144308 .loss/FRt1/detection_loss‚1.721201272600371 /loss/FRt2/detection_loss‚2.5995167841513953® —4Çžþ‚(:“ ¨õåÔÓ‰É gpu.0.gpu‚70.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚263.049' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.60979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÐ#ÿ‚(:˜ ¨õåÔÅÄïË( proc.memory.availableMB‚ 1765.6796875 disk.rbd1.out‚ 2.759765625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 1754.2265625 disk.nvme0n1.out‚ 50494.375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 214.640625 network.recv‚ 2.095971e+06 cpu‚25.305469758048364 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.637659072875977) disk./.usagePercent‚74.40288444303246 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 498.818359375" disk.nvme0c0n1.in‚ 1754.2265625% disk./.usageGB‚1108.6567459106445 disk.rbd1.in‚0 disk.dm-0.out‚ 50494.375 proc.cpu.threads‚17 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.sda3.out‚ 214.640625 network.sent‚ 1.6704057e+07 disk.sda2.in‚0 disk.dm-0.in‚ 1754.2265625" proc.memory.rssMB‚ 2725.9140625 disk.sda.out‚ 214.640625 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.nvme0c0n1.out‚ 50494.375$ memory_percent‚89.22314643859863 ¹»ž€‚(:“ ·õåÔçøÄÉ gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚253.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.772000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§'¤l3‚(:¨ ·õåÔ“¿ŒË disk.nvme0n1.in‚ 1754.2265625 network.sent‚ 1.6803556e+07 disk.rbd2.in‚0 network.recv‚ 2.10321e+06 disk.drbd1777.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚50494.70556640625* proc.memory.percent‚16.637659072875977 disk.dm-1.in‚0.5625 disk.sda1.in‚0 disk.sda2.out‚0# memory_percent‚89.2225980758667 cpu‚25.304771290161366 proc.cpu.threads‚17 disk.sda3.out‚215.3125" disk.nvme0c0n1.in‚ 1754.2265625 disk.sda.out‚215.3125 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.2265625 disk.sda2.in‚0) proc.memory.availableMB‚ 1765.76953125) disk./.usagePercent‚74.40289647538657( disk.nvme0c0n1.out‚50494.70556640625" proc.memory.rssMB‚ 2725.9140625 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.5625$ disk./.usageGB‚1108.656925201416 disk.rbd0.out‚0# disk.dm-0.out‚50494.70556640625 disk.dm-1.out‚215.3125 disk.rbd1.in‚0 disk.rbd2.out‚ 498.818359375 disk.rbd3.in‚0̪Tè2‚- _wandbruntime‚5768  _runtime‚5768 ÆÐ\ǃÁ _runtime‚5768.685793064 _step‚559 & lr-AdamW/pg1‚0.0001826027734081709 & lr-AdamW/pg2‚0.0001826027734081709 trainer/global_step‚6849 % _timestamp‚1.7884433286411664e+09úï‘ýÌ„Æ & lr-AdamW/pg1‚0.0001826027734081709 & lr-AdamW/pg2‚0.0001826027734081709 trainer/global_step‚6849 % _timestamp‚1.7884433286411664e+09 _runtime‚5768.685793064 _step‚559¯7k>ò2…- _wandbruntime‚5768  _runtime‚5768C§u(†ŠP RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚775360454119936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚652437604656128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚54785024 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 % _timestamp‚1.7884433286418538e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚652437632790528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127638678 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762656473883648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚750521074873344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12703953844736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47456407 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 _runtime‚5768.686716945 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 trainer/global_step‚6849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228594439 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100955761 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚775360454119936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100955767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65492900 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚750521049445376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12682906342522 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112949310 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚775357827518976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47455911 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚762656473883648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚750521074873344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12703953844736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47456407 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 _runtime‚5768.686716945 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 trainer/global_step‚6849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228594439 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100955761 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚775360454119936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100955767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65492900 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚750521049445376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12682906342522 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 ZLDeviceStatsMo'ªÙA]nitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112949310 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚775357827518976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47455911 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚652437632790528°e”ž2ˆ- _wandbruntime‚5769  _runtime‚5769fç'‰”N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228646975 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚775470226167296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12703185964032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112965024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65502096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12682137400522 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 % _timestamp‚1.7884433293941972e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762764413588992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47462472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669942516237312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚750627416047616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚762764439980544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763312151733014 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17412495393280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 trainer/global_step‚6849 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652529992709632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12684735685398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12705786186752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100969997 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47462925 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228626909 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚775467599553024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669942435428352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17412442718720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100970003 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12703185964032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127656912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127676972 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228626909 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚762764413588992 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127676972 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚561 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5769.439265984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112964568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552»;¢•'Š™N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚750627390619648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚762764413588992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47462472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669942516237312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚750627416047616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1574 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚762764439980544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763312151733014 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17412495393280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 trainer/global_step‚6849 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652529992709632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12684735685398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12705786186752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100969997 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47462925 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228626909 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚775467599553024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669942435428352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17412442718720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100970003 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12703185964032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127656912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127676972 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228626909 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚762764413588992 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127676972 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚561 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5769.439265984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112964568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228646975 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚775470226167296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12703185964032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112965024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65502096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12682137400522 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 % _timestamp‚1.7884433293941972e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 FC”&or.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101667567 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755838108158976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17534672365056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12774387067130 trainer/global_step‚6899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚768053459150848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65952703 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128553730 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚755838133586944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128553730 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47790258 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657057007331328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12792984484352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17534617580032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 % _timestamp‚1.7884433677376277e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101667561 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101667567 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚780846443635200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124n½(™P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47790748 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674591707830784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101667567 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755838108158976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17534672365056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12774387067130 trainer/global_step‚6899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚768053459150848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65952703 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128553730 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚755838133586944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128553730 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47790258 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657057007331328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12792984484352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17534617580032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 % _timestamp‚1.7884433677376277e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101667561 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101667567 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚780846443635200 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128592024 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113759279 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65961896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101681803 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚768161425247744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65961899 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 trainer/global_step‚6899 _step‚565 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚780958842283520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113758807 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101681803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47797380 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657149395384832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101681797 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128571964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657149423519232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5808.512889967 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17537118169088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12797417035776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768718067458378 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884433684675512e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚674686596460032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113758807 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3300917248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101681803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47797380 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657149395384832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101681797 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128571964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657149423519232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5808.512889967 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17537118169088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12797417035776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768718067458378 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884433684675512e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚674686596460032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128592024 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113759279 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65961896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101681803 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚768161425247744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65961899 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 trainer/global_step‚6899 _step‚565µi'Ǽ2- _wandbruntime‚5808  _runtime‚5808‚£¨ZžÔ /loss/FRt1/assignment_loss‚1.422344920039177 1loss/FBt4/assignment_loss‚0.09219562750917652 /loss/FRt1/detection_loss‚2.1693905813619496 1loss/SRqqt2/detection_loss‚1.9255471900105476 /loss/FBt4/detection_loss‚2.7196231285730996 1loss/FBt3/assignment_loss‚0.04922039985346297 0loss/SRqqt4/detection_loss‚1.988438490491647 /loss/FBt1/detection_loss‚0.3735527049209322 /loss/FRt4/assignment_loss‚3.094941406779819 2loss/SRqqt1/assignment_loss‚1.0006179812253784 /loss/FRt3/assignment_loss‚3.403975528639716 2loss/SRqqt2/assignment_loss‚1.2998963491144506 2loss/SRqqt3/assignment_loss‚1.3105783989438071 .loss/FBt2/detection_loss‚2.400256508588791 % _timestamp‚1.7884433684694297e+09 0loss/FBt1/assignment_loss‚0.1526117428875471 /loss/FRt3/detection_loss‚2.2590464318597476 1loss/SRqqt1/detection_loss‚1.1022427090901439 %loss/total_loss‚41.87064025567046 epoch‚5 /loss/FRt4/detection_loss‚2.4412214279174806 .loss/FBt3/detection_loss‚3.070523738861084 1loss/FBt2/assignment_loss‚0.17969273303169758 _step‚566 0loss/FRt2/assignment_loss‚2.7646022215485573 2loss/SRqqt4/assignment_loss‚1.3889983648386521 .loss/FRt2/detection_loss‚2.588488119095564 0loss/SRqqt3/detection_loss‚2.672633550488032 trainer/global_step‚6899 _runtime‚5808.514306526#¶_ŸÙ 1loss/SRqqt2/detection_loss‚1.9255471900105476 /loss/FBt4/detection_loss‚2.7196231285730996 1loss/FBt3/assignment_loss‚0.04922039985346297 0loss/SRqqt4/detection_loss‚1.988438490491647 /loss/FBt1/detection_loss‚0.3735527049209322 /loss/FRt4/assignment_loss‚3.094941406779819 2loss/SRqqt1/assignment_loss‚1.0006179812253784 /loss/FRt3/assignment_loss‚3.403975528639716 2loss/SRqqt2/assignment_loss‚1.2998963491144506 2loss/SRqqt3/assignment_loss‚1.3105783989438071 .loss/FBt2/detection_loss‚2.400256508588791 % _timestamp‚1.7884433684694297e+09 0loss/FBt1/assignment_loss‚0.1526117428875471 /loss/FRt3/detection_loss‚2.2590464318597476 1loss/SRqqt1/detection_loss‚1.1022427090901439 %loss/total_loss‚41.87064025567046 epoch‚5 /loss/FRt4/detection_loss‚2.4412214279174806 .loss/FBt3/detection_loss‚3.070523738861084 1loss/FBt2/assignment_loss‚0.17969273303169758 _step‚566 0loss/FRt2/assignment_loss‚2.7646022215485573 2loss/SRqqt4/assignment_loss‚1.3889983648386521 .loss/FRt2/detection_loss‚2.588488119095564 0loss/SRqqt3/detection_loss‚2.672633550488032 trainer/global_step‚6899 _runtime‚5808.514306526 /loss/FRt1/assignment_loss‚1.422344920039177 1loss/FBt4/assignment_loss‚0.09219562750917652 /loss/FRt1/detection_loss‚2.1693905813619496¶ªµ”០‚(:” óõåÔ†¸É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚251.622' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.324400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çRe=¡‚(:² óõåÔ›àùË disk.rbd3.out‚0& disk.nvme0n1.out‚50691.17333984375 disk.sda3.in‚0.5625 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚74.40292694028315 disk.rbd2.in‚0 disk.dm-1.out‚ 217.171875 disk.sda.out‚ 217.171875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2725.9140625 disk.sda2.in‚0% disk./.usageGB‚1108.6573791503906 disk.rbd1.in‚0 disk.rbd2.out‚ 502.134765625( disk.nvme0c0n1.out‚50691.17333984375 disk.drbd1777.in‚0* proc.memory.percent‚16.637659072875977 cpu‚25.302149293655123! disk.nvme0n1.in‚ 1754.23046875 disk.dm-1.in‚0.5625$ memory_percent‚89.22674655914307 proc.cpu.threads‚17 network.sent‚ 1.693069e+07 disk.rbd3.in‚0 disk.rbd0.out‚0 network.recv‚ 2.12262e+06 disk.sda3.out‚ 217.171875# disk.nvme0c0n1.in‚ 1754.23046875 disk.dm-0.in‚ 1754.23046875# disk.dm-0.out‚50691.17333984375) proc.memory.availableMB‚ 1765.08984375 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.5625 6+”¢‚(:‰ ‚öåÔ¿ÊÀÉ gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚44.0 gpu.0.powerWatts‚103.502' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ϘµÚC£‚(:¸ ‚öåÔö؉Ë disk.rbd0.in‚0 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2725.9140625) disk./.usagePercent‚74.40292719629068* proc.memory.percent‚16.637659072875977 disk.dm-0.in‚ 1754.23046875 disk.sda.out‚ 217.48828125 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1765.62109375 cpu‚25.30146919309573 disk.sda3.out‚ 217.48828125 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 1.7040037e+07 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 503.767578125$ memory_percent‚89.22350406646729 proc.cpu.threads‚17 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1754.23046875 disk.rbd1.out‚ 2.759765625 disk.rbd2.in‚0$ disk./.usageGB‚1108.657382965088& disk.nvme0n1.out‚50691.65380859375# disk.dm-0.out‚50691.65380859375 disk.dm-1.out‚ 217.48828125 disk.rbd1.in‚0 network.recv‚ 2.129208e+06! disk.nvme0n1.in‚ 1754.23046875( disk.nvme0c0n1.out‚50691.65380859375 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.sda3.in‚0.5625} h2¤- _wandbruntime‚5846  _runtime‚5846uåvèȥ ' lr-AdamW/pg1‚0.00017702326828231404 ' lr-AdamW/pg2‚0.00017702326828231404 trainer/global_step‚6949 % _timestamp‚1.7884434065427525e+09 _runtime‚ 5846.58742288 _step‚567 ‚½Í¦Ç trainer/global_step‚6949 % _timestamp‚1.7884434065427525e+09 _runtime‚ 5846.58742288 _step‚567 ' lr-AdamW/pg1‚0.00017702326828231404 ' lr-AdamW/pg2‚0.00017702326828231404· ×…"2§- _wandbruntime‚5846  _runtime‚5846zMtÖ(¨ŠP TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66412500 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚774021060092282 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48124373 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786337686352384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786335059751424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12887215542784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102379367 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773450470809600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66412503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚679335773240320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12865867791738 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786335059751424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102379361 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚6949 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102379361 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17659363233792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761155192300544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661676438140928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129468782 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12884615333376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12863269507070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773450444418048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3300917248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129468782 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12884615333376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12863269507070 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773450444418048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66412503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚679335773240320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12865867791738 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786335059751424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102379361 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚6949 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102379361 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17659363233792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚761155192300544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661676438140928 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6949 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12886447662080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _runtime‚ 5847.32128144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114552690 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚774126606896586 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12865098849738 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129507076 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66421696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12889047884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚786444831785472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773558384123392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17661880585728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚773558410514944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚761261533474816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 _step‚569 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48131456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661768826194432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚774129230609430 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231900679 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚761261508046848 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12889047884800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚679430759454720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12867697134614 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102393603 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773558410514944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚679430678645760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚786447458399744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚786447458399744 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114553155 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17661933260288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚254279685Ys'¬˜N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661768826194432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚774129230609430 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231900679 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚761261508046848 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12889047884800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚679430759454720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12867697134614 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102393603 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773558410514944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚679430678645760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚786447458399744 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚786447458399744 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23670554624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114553155 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17661933260288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12886447662080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102393603 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6949 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12886447662080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _runtime‚ 5847.32128144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114552690 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚774126606896586 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12865098849738 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129507076 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66421696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12889047884800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚786444831785472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773558384123392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17661880585728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23670554624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚773558410514944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1574 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚761261533474816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3300917248 _step‚569 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48131456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3300917248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552¹ éÃT2­- _wandbruntime‚5847  _runtime‚5847eDÑ]®× % _timestamp‚1.7884434072778416e+09 /loss/FRt1/detection_loss‚1.6465550585091113 1loss/SRqqt3/detection_loss‚2.3178083336714543 /loss/FBt1/detection_loss‚0.6360537941975053 _runtime‚5847.322397584 1loss/SRqqt1/detection_loss‚1.2493954435235164 .loss/FBt2/detection_loss‚2.310813163008009 .loss/FBt4/detection_loss‚2.678056021531423 0loss/FRt3/assignment_loss‚3.4402105916630137 /loss/FRt4/assignment_loss‚2.429135760775319 2loss/SRqqt1/assignment_loss‚0.9488693392214358 1loss/FBt1/assignment_loss‚0.06889138106707629 .loss/FRt2/detection_loss‚2.415688974871522 0loss/FRt2/assignment_loss‚3.1426122656890323 1loss/SRqqt4/detection_loss‚1.7777235714613266 2loss/SRqqt4/assignment_loss‚1.4028926210583381 epoch‚5 _step‚570 trainer/global_step‚6949 2loss/SRqqt2/assignment_loss‚1.3451228467530982 2loss/SRqqt3/assignment_loss‚1.8376633272026524 1loss/FBt4/assignment_loss‚0.07548695972794424 /loss/FBt3/detection_loss‚3.0080182030797005 %loss/total_loss‚41.68124981511166 /loss/FRt4/detection_loss‚2.0793844908475876 1loss/FBt2/assignment_loss‚0.29540841547506197 0loss/FBt3/assignment_loss‚0.2819559607087285 .loss/FRt3/detection_loss‚3.315435120299536 0loss/FRt1/assignment_loss‚1.1301931190490722 1loss/SRqqt2/detection_loss‚1.8478750517201978€oÁÇb¯Ü 2loss/SRqqt4/assignment_loss‚1.4028926210583381 epoch‚5 _step‚570 trainer/global_step‚6949 2loss/SRqqt2/assignment_loss‚1.3451228467530982 2loss/SRqqt3/assignment_loss‚1.8376633272026524 1loss/FBt4/assignment_loss‚0.07548695972794424 /loss/FBt3/detection_loss‚3.0080182030797005 %loss/total_loss‚41.68124981511166 /loss/FRt4/detection_loss‚2.0793844908475876 1loss/FBt2/assignment_loss‚0.29540841547506197 0loss/FBt3/assignment_loss‚0.2819559607087285 .loss/FRt3/detection_loss‚3.315435120299536 0loss/FRt1/assignment_loss‚1.1301931190490722 1loss/SRqqt2/detection_loss‚1.8478750517201978 % _timestamp‚1.7884434072778416e+09 /loss/FRt1/detection_loss‚1.6465550585091113 1loss/SRqqt3/detection_loss‚2.3178083336714543 /loss/FBt1/detection_loss‚0.6360537941975053 _runtime‚5847.322397584 1loss/SRqqt1/detection_loss‚1.2493954435235164 .loss/FBt2/detection_loss‚2.310813163008009 .loss/FBt4/detection_loss‚2.678056021531423 0loss/FRt3/assignment_loss‚3.4402105916630137 /loss/FRt4/assignment_loss‚2.429135760775319 2loss/SRqqt1/assignment_loss‚0.9488693392214358 1loss/FBt1/assignment_loss‚0.06889138106707629 .loss/FRt2/detection_loss‚2.415688974871522 0loss/FRt2/assignment_loss‚3.1426122656890323 1loss/SRqqt4/detection_loss‚1.7777235714613266º1õöŠ“°‚(:ˆ ‘öåÔïÀ›É gpu.0.gpu‚88.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚271.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.257 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p‡ C±‚(:¸ ‘öåÔîì…Ì disk.sda1.in‚0 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚50789.10693359375 disk.dm-1.in‚0.5625$ memory_percent‚89.22488689422607 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1754.23046875 disk.drbd1777.in‚0 disk.rbd2.in‚0& disk.nvme0n1.out‚50789.10693359375 disk.rbd0.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda1.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚50789.10693359375 cpu‚25.30077621207929 disk.rbd3.in‚0 disk.sda3.out‚ 217.97265625 disk.dm-0.in‚ 1754.23046875 disk.rbd1.in‚0( disk./.usagePercent‚74.4029310364037% disk./.usageGB‚1108.6574401855469 disk.sda.in‚0.5625 disk.sda.out‚ 217.97265625 disk.rbd3.out‚0 disk.rbd2.out‚ 505.279296875 network.sent‚ 1.7045971e+07 disk.sda2.in‚0 disk.drbd1777.out‚0) proc.memory.availableMB‚ 1765.39453125" proc.memory.rssMB‚ 2725.9140625! disk.nvme0n1.in‚ 1754.23046875 network.recv‚ 2.133638e+06 disk.dm-1.out‚ 217.97265625* proc.memory.percent‚16.637659072875977 proc.cpu.threads‚17ŠÖ«”²‚(:‰  öåÔ¦¡ÑÈ gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚231.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hy J7³‚(:¬  öåÔ¬‰ÑË disk.rbd1.out‚ 2.759765625 disk.dm-1.in‚0.5625* proc.memory.percent‚16.637659072875977 cpu‚25.300141474569116 disk.rbd0.out‚0 disk.dm-1.out‚ 218.4765625 disk.drbd1777.in‚0! disk.nvme0n1.in‚ 1754.23046875" proc.memory.rssMB‚ 2725.9140625 disk.rbd2.in‚0 disk.rbd3.in‚0" disk.nvme0n1.out‚ 50790.0703125 disk.sda3.in‚0.5625 disk.sda3.out‚ 218.4765625 disk.rbd0.in‚0) proc.memory.availableMB‚ 1765.37890625 network.recv‚ 2.140656e+06) disk./.usagePercent‚74.40292719629068$ disk./.usageGB‚1108.657382965088 disk.drbd1777.out‚0! disk.rbd2.out‚505.45947265625$ disk.nvme0c0n1.out‚ 50790.0703125 disk.rbd1.in‚0 disk.sda.out‚ 218.4765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1754.23046875 disk.dm-0.out‚ 50790.0703125$ memory_percent‚89.22498226165771 disk.sda.in‚0.5625 disk.dm-0.in‚ 1754.23046875 proc.cpu.threads‚17 network.sent‚ 1.7155725e+07 disk.sda2.in‚0 disk.sda2.out‚0Þºà”´‚(:‰ ¯öåÔ™—É gpu.0.gpu‚71.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚48.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}úm(8µ‚(:­ ¯öåÔŸÈûÊ disk.dm-1.in‚0.5625 disk.dm-1.out‚ 218.90625' disk.nvme0c0n1.out‚50790.4208984375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625# memory_percent‚89.2265796661377 cpu‚25.29951215110384) disk./.usagePercent‚74.40293231644137 disk.rbd0.out‚0 disk.sda.out‚ 218.90625 disk.rbd3.out‚0" proc.memory.rssMB‚ 2725.9140625 disk.dm-0.in‚ 1754.23046875 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.145305e+06 disk.rbd0.in‚0! disk.rbd2.out‚506.97119140625 disk.sda3.out‚ 218.90625( proc.memory.availableMB‚ 1765.1171875 disk.rbd1.out‚ 2.759765625 disk.sda.in‚0.5625 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.23046875 network.sent‚ 1.7161961e+07 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1754.23046875" disk.dm-0.out‚50790.4208984375% disk.nvme0n1.out‚50790.4208984375* proc.memory.percent‚16.637659072875977 disk.drbd1777.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1108.6574592590332Öz|˜2¶- _wandbruntime‚5885  _runtime‚5885ú,yFÇ·Á & lr-AdamW/pg2‚0.0001742384561222857 trainer/global_step‚6999 % _timestamp‚1.7884434456208916e+09 _runtime‚5885.665496715 _step‚571 & lr-AdamW/pg1‚0.0001742384561222857gbzDÌ¸Æ & lr-AdamW/pg1‚0.0001742384561222857 & lr-AdamW/pg2‚0.0001742384561222857 trainer/global_step‚6999 % _timestamp‚1.7884434456208916e+09 _runtime‚5885.665496715 _step‚571»3Öñ#2¹- _wandbruntime‚5885  _runtime‚5885ÈÀ`Éo ºŠP TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12976246182400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚791826302468608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17784264856576 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _step‚572 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130403868 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130383834 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚778847429685248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚684080024850432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 _runtime‚5885.666514032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103091161 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66872303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884434456216097e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79LNèË¡1823675867648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚779429599530490 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233495035 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚766472225586176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚500 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 trainer/global_step‚6999 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚766472251014144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103091167 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚778847456076800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778847456076800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115332443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48460640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚791823675867648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚778847429685248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115332943 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12978846391808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12957348516346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12978846391808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12954750231678 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648Æ^[Ã(»P `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚766472251014144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103091167 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚778847456076800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚778847456076800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115332443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48460640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1574 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚791823675867648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚778847429685248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115332943 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12978846391808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12957348516346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12978846391808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12954750231678 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12976246182400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚791826302468608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17784264856576 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1574 _step‚572 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130403868 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23670554624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130383834 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚778847429685248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23670554624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚684080024850432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1574 _runtime‚5885.666514032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103091161 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66872303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23670554624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884434456216097e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚791823675867648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚779429599530490 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233495035 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚766472225586176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚500 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3300917248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2648 trainer/global_step‚6999¼ßSúÊ2¼- _wandbruntime‚5886  _runtime‚58860çÄ'½”N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚778955369390592 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48467489 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚791936074515968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12980678733824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚778955395782144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130422128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚791936074515968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12978078511104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48466998 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666388228869632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚779535146334794 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66881499 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103105403 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚684174906120192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚494 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115348494 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12959177859222 % _timestamp‚1.7884434463595269e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12978078511104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12956579574346 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚6999 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130402068 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130402068 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12980678733824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115348988 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚778955395782144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233507465 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376ñ».PÛ¾™N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130402068 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130402068 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12980678733824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115348988 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚778955395782144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233507465 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚778955369390592 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48467489 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚791936074515968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12980678733824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚778955395782144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130422128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚791936074515968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12978078511104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48466998 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23670554624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666388228869632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚779535146334794 PADeviceStatsMonitor.on_train_batch_end/allocate‹ÝòDd_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66881499 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103105403 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚684174906120192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚494 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115348494 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12959177859222 % _timestamp‚1.7884434463595269e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12978078511104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12956579574346 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚6999 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376½«$c€2¿- _wandbruntime‚5886  _runtime‚5886¹ýbYÀÓ /loss/FRt4/assignment_loss‚2.918551340335753 2loss/SRqqt3/assignment_loss‚1.5666843865066766 2loss/SRqqt4/assignment_loss‚1.4126638532534503 1loss/SRqqt3/detection_loss‚2.0481331676244734 _runtime‚5886.405723104 0loss/FRt1/assignment_loss‚1.0648343932628632 1loss/FBt2/assignment_loss‚0.31697386913001535 /loss/FBt4/detection_loss‚2.5861716747283934 _step‚574 % _timestamp‚1.7884434463611193e+09 /loss/FRt2/assignment_loss‚2.397053541077508 /loss/FRt3/assignment_loss‚3.504823444570814 /loss/FRt2/detection_loss‚2.3793497376971775 /loss/FRt4/detection_loss‚2.3427075939934427 0loss/SRqqt4/detection_loss‚1.902866445612489 trainer/global_step‚6999 /loss/FBt3/assignment_loss‚0.444570195106644 /loss/FRt1/detection_loss‚1.4337999311089515 2loss/SRqqt2/assignment_loss‚1.4366420281585306 .loss/SRqqt2/detection_loss‚1.9725459505183 epoch‚5 2loss/SRqqt1/assignment_loss‚0.8955153883900493 1loss/FBt1/assignment_loss‚0.18474916697491822 /loss/FBt2/detection_loss‚2.7154566049575806 /loss/FBt3/detection_loss‚2.7542332887649534 &loss/total_loss‚40.395630109200084 0loss/FBt4/assignment_loss‚0.4803253636695445 -loss/FRt3/detection_loss‚2.40385820377352 1loss/SRqqt1/detection_loss‚0.7480431850068271 0loss/FBt1/detection_loss‚0.48507735497721083Рë^ÁØ 2loss/SRqqt1/assignment_loss‚0.8955153883900493 1loss/FBt1/assignment_loss‚0.18474916697491822 /loss/FBt2/detection_loss‚2.7154566049575806 /loss/FBt3/detection_loss‚2.7542332887649534 &loss/total_loss‚40.395630109200084 0loss/FBt4/assignment_loss‚0.4803253636695445 -loss/FRt3/detection_loss‚2.40385820377352 1loss/SRqqt1/detection_loss‚0.7480431850068271 0loss/FBt1/detection_loss‚0.48507735497721083 /loss/FRt4/assignment_loss‚2.918551340335753 2loss/SRqqt3/assignment_loss‚1.5666843865066766 2loss/SRqqt4/assignment_loss‚1.4126638532534503 1loss/SRqqt3/detection_loss‚2.0481331676244734 _runtime‚5886.405723104 0loss/FRt1/assignment_loss‚1.0648343932628632 1loss/FBt2/assignment_loss‚0.31697386913001535 /loss/FBt4/detection_loss‚2.5861716747283934 _step‚574 % _timestamp‚1.7884434463611193e+09 /loss/FRt2/assignment_loss‚2.397053541077508 /loss/FRt3/assignment_loss‚3.504823444570814 /loss/FRt2/detection_loss‚2.3793497376971775 /loss/FRt4/detection_loss‚2.3427075939934427 0loss/SRqqt4/detection_loss‚1.902866445612489 trainer/global_step‚6999 /loss/FBt3/assignment_loss‚0.444570195106644 /loss/FRt1/detection_loss‚1.4337999311089515 2loss/SRqqt2/assignment_loss‚1.4366420281585306 .loss/SRqqt2/detection_loss‚1.9725459505183 epoch‚5¾è¶ß>”‚(:‰ ¾öåÔº¢£Ê gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚45.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x:¹›BÂ(:· ¾öåÔàþ¹Ë network.recv‚ 2.148772e+06 disk.rbd3.out‚0 disk.dm-1.in‚0.5625' proc.memory.availableMB‚ 1765.390625! disk.nvme0n1.in‚ 1754.23046875" proc.memory.rssMB‚ 2725.9140625* proc.memory.percent‚16.637659072875977 cpu‚25.298875743737376 proc.cpu.threads‚17( disk.nvme0c0n1.out‚50888.31396484375) disk./.usagePercent‚74.40293001237356 disk.sda.out‚ 219.2890625# disk.nvme0c0n1.in‚ 1754.23046875 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.23046875 disk.sda.in‚0.5625 disk.sda1.out‚0& disk.nvme0n1.out‚50888.31396484375% disk./.usageGB‚1108.6574249267578 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0! disk.rbd2.out‚507.07666015625# disk.dm-0.out‚50888.31396484375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0$ memory_percent‚89.22491073608398 network.sent‚ 1.7167163e+07 disk.rbd1.out‚ 2.759765625 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 219.2890625 disk.dm-1.out‚ 219.2890625Æ’%ŸÄ‚(:” ÍöåÔ¾ô‘É gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.195877075195312* gpu.0.memoryAllocatedBytes‚ 24220073984 gpu.0.temp‚47.0 gpu.0.powerWatts‚274.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.948600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ÍË]BÅ‚(:· ÍöåÔÝ”ŸË$ memory_percent‚89.22581672668457 cpu‚25.29824881127914 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.40293154841878 disk.rbd2.in‚0 network.sent‚ 1.7276962e+07 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2725.9140625& disk.nvme0n1.out‚50888.62255859375 disk.dm-1.out‚ 219.7734375 disk.rbd1.out‚ 2.845703125 disk.sda.out‚ 219.7734375 network.recv‚ 2.155646e+06 disk.sda3.out‚ 219.7734375 disk.rbd1.in‚0 disk.sda.in‚0.5625* proc.memory.percent‚16.637659072875977% disk./.usageGB‚1108.6574478149414# disk.nvme0c0n1.in‚ 1754.23046875! disk.rbd2.out‚508.58837890625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1754.23046875# disk.dm-0.out‚50888.62255859375 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1765.2421875! disk.nvme0n1.in‚ 1754.23046875( disk.nvme0c0n1.out‚50888.62255859375 disk.drbd1777.out‚0 disk.sda2.in‚0†òÚ2Æ- _wandbruntime‚5924  _runtime‚5924.íûBÉÇà ' lr-AdamW/pg1‚0.00017145771160734656 ' lr-AdamW/pg2‚0.00017145771160734656 trainer/global_step‚7049 % _timestamp‚1.7884434844318974e+09 _runtime‚5924.476655219 _step‚575æ­ JÎÈÈ ' lr-AdamW/pg1‚0.00017145771160734656 ' lr-AdamW/pg2‚0.00017145771160734656 trainer/global_step‚7049 % _timestamp‚1.7884434844318974e+09 _runtime‚5924.476655219 _step‚575¿d*a2É- _wandbruntime‚5924  _runtime‚5924„Ÿ'(ʉP trainer/global_step‚7049 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 $ _timestamp‚1.788443484432685e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131298886 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797312291983872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103802961 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103802961 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13048829240954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13067877031424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17909199145984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚797312291983872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103802967 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚670915243491328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13070477240832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚784244441344000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131298886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797314918584832 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116127467 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚784244414952448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103802967 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚771789309727744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13067877031424 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116126965 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688824442637312 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _step‚576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚784244441344000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67332103 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131318920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚784244414952448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48795364 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚670915215356928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67332100 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575Ÿ!ÙÃ(ËŽP TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131298886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚797314918584832 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116127467 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚784244414952448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103802967 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚771789309727744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13067877031424 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116126965 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688824442637312 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _step‚576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚784244441344000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67332103 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131318920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚784244414952448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48795364 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚670915215356928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67332100 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 trainer/global_step‚7049 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 $ _timestamp‚1.788443484432685e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131298886 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797312291983872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103802961 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103802961 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13048829240954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13067877031424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17909199145984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚797312291983872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103802967 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚670915243491328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13070477240832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚784244441344000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376Àä8GÌ‚(:’ ÜöåÔ“­‰É gpu.0.gpu‚72.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚249.2' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.839999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$1"ñÍ‚(:¹ ÜöåÔÊòÊ disk.sda2.in‚0 network.sent‚ 1.7282722e+07 disk.rbd2.in‚0 network.recv‚ 2.160128e+06 disk.dm-1.out‚ 220.46484375 disk.sda2.out‚0 disk.rbd3.in‚0* proc.memory.percent‚16.637659072875977 disk.dm-1.in‚0.5625 disk.sda.out‚ 220.46484375% disk./.usageGB‚1108.6574478149414! disk.nvme0n1.in‚ 1754.23046875 disk.sda3.in‚0.5625 disk.sda3.out‚ 220.46484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2725.9140625 cpu‚25.2976262616094 disk.drbd1777.in‚0! disk.rbd2.out‚510.33056640625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1754.23046875 disk.rbd1.in‚0$ memory pi&S_percent‚89.22572135925293( proc.memory.availableMB‚ 1765.2578125 disk.rbd1.out‚ 2.908203125# disk.dm-0.out‚50986.11474609375) disk./.usagePercent‚74.40293154841878& disk.nvme0n1.out‚50986.11474609375( disk.nvme0c0n1.out‚50986.11474609375 disk.dm-0.in‚ 1754.23046875 disk.sda.in‚0.5625 disk.sda1.in‚0Oàui2Î- _wandbruntime‚5925  _runtime‚5925ÈZÜq'Ï”N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13072309582848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13069709360128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116143246 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17911701514240 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚784352354657792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚671007603410432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671007631544832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131337180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚5925.208334148 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48801947 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235154383 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131317120 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48801463 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚771895625474048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚784943685773002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67341296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚771895650902016 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚797424690632192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚784352381049344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13069709360128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116142759 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103817203 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13050658583830 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688919250152960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚577 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103817197 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67341299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚784946309485846 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚784352381049344™$2F'ЙN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116142759 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103817203 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13050658583830 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688919250152960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚577 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103817197 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67341299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚784946309485846 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚784352381049344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13072309582848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13069709360128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116143246 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17911701514240 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚784352354657792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚671007603410432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671007631544832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131337180 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚5925.208334148 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48801947 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235154383 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131317120 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48801463 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚771895625474048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚784943685773002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67341296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚771895650902016 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚797424690632192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚784352381049344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13069709360128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E]¾ÎÚÈ ' lr-AdamW/pg1‚0.00016868161424048577 ' lr-AdamW/pg2‚0.00016868161424048577 trainer/global_step‚7099 % _timestamp‚1.7884435235090454e+09 _runtime‚5963.553701042 _step‚579Ãñ\´52Û- _wandbruntime‚5963  _runtime‚5963‹°¯È;܉P H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚789641400219648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13140309965562 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚501 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777106343013376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104514767 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104514761 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132213938 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚789641426611200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13137711680894 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67791903 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚789641400219648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚802803534701056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _runtime‚5963.554694955 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132233972 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚790244054694270 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚790246678406906 _step‚580 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚802800908100096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675534618032128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚7099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_byte/¥9òÔs.all.allocated‚802803534701056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67791900 $ _timestamp‚1.788443523509796e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚789641426611200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13162108089856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236728699 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116920759 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777106368441344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116920255 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚54785024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13159507880448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104514767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49128355 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675534646166528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104514761 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132213938 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18034152530432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777106368441344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116920255 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚54785024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13159507880448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104514767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49128355 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675534646166528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104514761 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132213938 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18034152530432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚789641400219648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13140309965562 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚501 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777106343013376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104514767 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104514761 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132213938 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚789641426611200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚82919424 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13137711680894 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67791903 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚789641400219648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚802803534701056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _runtime‚5963.554694955 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132233972 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚790244054694270 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚790246678406906 _step‚580 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚802800908100096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184Ä]ö?•2Þ- _wandbruntime‚5964  _runtime‚5964´R/'ß”N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104529003 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116936106 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 _runtime‚5964.289741871 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚789749339924992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790354848924054 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675627034220032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67801099 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132252232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 % _timestamp‚1.7884435242446496e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚7099 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802913306748416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132232172 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18036593246208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13139541023562 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚789749366316544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚789749339924992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802910680134144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚484 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚693663682237952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13163940431872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67801096 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777212709615616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚581 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚789749366316544 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13161340209152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132232172 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104528997 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104528997 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49135010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104529003 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802913306748416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13163940431872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236761169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132252232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18036648017920 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49135494 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693663599331840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚790352225211210 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876+ôP/©à™N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802910680134144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚484 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚693663682237952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13163940431872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67801096 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777212709615616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚581 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚789749366316544 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13161340209152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132232172 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104528997 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104528997 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49135010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104529003 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802913306748416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13163940431872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236761169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132252232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18036648017920 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49135494 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693663599331840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚790352225211210 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104529003 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116936106 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 _runtime‚5964.289741871 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚789749339924992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790354848924054 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675627034220032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67801099 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132252232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.6hò,vpeak‚1575 % _timestamp‚1.7884435242446496e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 trainer/global_step‚7099 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802913306748416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132232172 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18036593246208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13139541023562 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚789749366316544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚789749339924992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚808289524216320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13251138729472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133149024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68251703 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚698312995296256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 % _timestamp‚1.7884435622978911e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133128990 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133149024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105226567 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚795038385486848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 _runtime‚6002.342845577 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105226561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚795038411878400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13231790690170 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚808292150817280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚808292150817280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105226561 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚782423427154944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133128990 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795655217845114 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117716403 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚698313076118528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚584 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚795038411878400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13251138729472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚680154048841728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13229192405502 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238355551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795652594132478 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚782423401726976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚485 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0g…d(ïP `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚782423427154944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133128990 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795655217845114 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117716403 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚698313076118528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚584 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚795038411878400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13251138729472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚680154048841728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13229192405502 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238355551 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚795652594132478 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚782423401726976 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚485 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚808289524216320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13251138729472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133149024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68251703 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚698312995296256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 % _timestamp‚1.7884435622978911e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133128990 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133149024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105226567 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚795038385486848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 _runtime‚6002.342845577 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105226561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚795038411878400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13231790690170 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚808292150817280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚808292150817280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105226561ÈüµIQ2ð- _wandbruntime‚6003  _runtime‚6003Àm÷TB ñ“N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13231021748170 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133167284 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚795760764649418 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚795146325192192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚808399296250368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18161523857408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚795146351583744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.smalldÛ×_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133167284 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105240797 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚680246408760832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚585 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13252971058176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚698407879943680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚808401922864640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚808401922864640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚7149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133147224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13233620033046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788443563031591e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105240803 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13255571280896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238388021 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117732216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49471320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚680246436895232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117732680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49471781 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚782529768329216 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105240797 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚680246408760832 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚585 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13252971058176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚698407879943680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚808401922864640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚808401922864640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚7149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133147224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13233620033046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788443563031591e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105240803 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13255571280896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238388021 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117732216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49471320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚680246436895232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117732680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49471781 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚782529768329216 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18161523857408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚795146351583744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133167284 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13345369787904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚800435397145600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚801061133570686 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105938367 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68711503 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚813778140332544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚800435397145600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703056981705216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49799523 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚800435370754048 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787740460440576 trainer/global_step‚7199 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚703057062527488 _runtime‚6041.362795114 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚486 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684773423382528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118511026 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68711500 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13320673130110 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118510540 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13323271414778 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚801063757283322 _step‚588 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚483 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18283558322688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134044042 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134064076 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚813778140332544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18283611010560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚813780766933504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134044042 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13342769578496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105938361 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.alloÅ; ôcated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚800435370754048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚787740485868544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49799040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788443601317945e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚813780766933504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134064076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575j1¯F(ÿŽP YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240002443 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49799523 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚800435370754048 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787740460440576 trainer/global_step‚7199 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚703057062527488 _runtime‚6041.362795114 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚486 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684773423382528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118511026 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68711500 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13320673130110 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118510540 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13323271414778 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚801063757283322 _step‚588 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚483 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18283558322688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134044042 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134064076 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚813778140332544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18283611010560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚813780766933504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134044042 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13342769578496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105938361 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚800435370754048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚787740485868544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49799040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.788443601317945e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚813780766933504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134064076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105938367 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13345369787904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13342769578496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13345369787904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚800435397145600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚801061133570686 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105938367 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68711503 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚813778140332544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚800435397145600 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703056981705216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ÌS³2€- _wandbruntime‚6042  _runtime‚6042ÇÐĨ'”N F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105952597 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134082336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13347202129920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18286057016832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13344601907200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49806024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703151868452864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684865811436032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 % _timestamp‚1.7884436020514548e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚800543310459392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240014873 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134082336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚800543336850944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13344601907200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 _runtime‚6042.096751614 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105952603 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105952603 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚813887912366592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚801171927800470 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118526261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787846801614848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68720696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134062276 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118526723 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13325100757654 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚800543310459392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684865839570432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13344601907200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49806024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚703151868452864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684865811436032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 % _timestamp‚1.7884436020514548e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚800543310459392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240014873 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134082336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚800543336850944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13344601907200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 _runtime‚6042.096751614 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105952603 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚82906112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105952603 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚813887912366592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚801171927800470 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118526261 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787846801614848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68720696 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134062276 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118526723 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13325100757654 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚800543310459392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684865839570432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚707801072519680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚819266756448768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚492 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18408163542528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119304267 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69171300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805832356021248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚689392854192128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134959094 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119303775 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805832356021248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134979128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50132475 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13412153854718 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134979128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106650167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚819269383049728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚707800989600256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13434400427520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18408218327552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106650161 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚819266756448768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚492 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18408163542528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119304267 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69171300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805832356021248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚689392854192128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134959094 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119303775 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805832356021248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134979128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50132475 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13412153854718 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134979128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106650167 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚819269383049728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚707800989600256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13434400427520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18408218327552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106650161 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚707801072519680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50139129 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚793163885756416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241641725 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13438832978944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106664403 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚806577843525834 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13436232756224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13436232756224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106664403 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚819376528482816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119320096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69180496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 $ _timestamp‚1.788443641145153e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚819379155097088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚689485214111232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚805940295726592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚805940322118144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134997388 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119319625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50139597 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18410643313152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69180499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚593 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 _runtime‚6081.190180074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106664397 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18410698084864 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707895857424384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13413983197386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50139129 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚793163885756416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241641725 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13438832978944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106664403 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚806577843525834 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13436232756224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13436232756224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106664403 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚819376528482816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119320096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69180496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 $ _timestamp‚1.788443641145153e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚819379155097088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚689485214111232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚805940295726592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚805940322118144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134997388 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119319625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50139597 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18410643313152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69180499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚593 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 _runtime‚6081.190180074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106664397 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18410698084864 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575ÑVH¹ä2•- _wandbruntime‚6081  _runtime‚6081lûs`–Ú _runtime‚6081.191287852 0loss/FRt4/assignment_loss‚2.6749709126624195 1loss/FBt3/assignment_loss‚0.15762152391419346 /loss/FBt2/detection_loss‚2.1510817855596542 .loss/FBt3/detection_loss‚3.589586819921221 1loss/FBt1/assignment_loss‚0.07090916066822102 /loss/FRt3/detection_loss‚2.8021040992153474 0loss/FRt1/assignment_loss‚1.0582964583939196 /loss/FRt4/detection_loss‚2.3679903495040806 /loss/FBt1/detection_loss‚0.5371393796347548 &loss/total_loss‚41.073806365691055 2loss/SRqqt2/assignment_loss‚1.2789649526588618 2loss/SRqqt3/assignment_loss‚1.7308294833637774 1loss/SRqqt2/detection_loss‚1.9258187964558602 .loss/FRt2/detection_loss‚2.413247518918731 % _timestamp‚1.7884436411467335e+09 1loss/SRqqt3/detection_loss‚2.5941481329500675 /loss/FBt4/detection_loss‚1.8899786216872079 epoch‚5 0loss/FRt2/assignment_loss‚2.7912016998637807 /loss/FRt3/assignment_loss‚3.698936306197068 2loss/SRqqt1/assignment_loss‚0.9881788171737248 2loss/SRqqt4/assignment_loss‚1.2929259308389034 0loss/FBt2/assignment_loss‚0.7146323060733266 /loss/FRt1/detection_loss‚1.5883606716113932 _step‚594 trainer/global_step‚7249 1loss/FBt4/assignment_loss‚0.12398215503032718 1loss/SRqqt1/detection_loss‚1.0025578367316887 1loss/SRqqt4/detection_loss‚1.6303426466625313Ò\Ze—ß 0loss/FRt2/assignment_loss‚2.7912016998637807 /loss/FRt3/assignment_loss‚3.698936306197068 2loss/SRqqt1/assignment_loss‚0.9881788171737248 2loss/SRqqt4/assignment_loss‚1.2929259308389034 0loss/FBt2/assignment_loss‚0.7146323060733266 /loss/FRt1/detection_loss‚1.5883606716113932 _step‚594 trainer/global_step‚7249 1loss/FBt4/assignment_loss‚0.12398215503032718 1loss/SRqqt1/detection_loss‚1.0025578367316887 1loss/SRqqt4/detection_loss‚1.6303426466625313 _runtime‚6081.191287852 0loss/FRt4/assignment_loss‚2.6749709126624195 1loss/FBt3/assignment_loss‚0.15762152391419346 /loss/FBt2/detection_loss‚2.1510817855596542 .loss/FBt3/detection_loss‚3.589586819921221 1loss/FBt1/assignment_loss‚0.07090916066822102 /loss/FRt3/detection_loss‚2.8021040992153474 0loss/FRt1/assignment_loss‚1.0582964583939196 /loss/FRt4/detection_loss‚2.3679903495040806 /loss/FBt1/detection_loss‚0.5371393796347548 &loss/total_loss‚41.073806365691055 2loss/SRqqt2/assignment_loss‚1.2789649526588618 2loss/SRqqt3/assignment_loss‚1.7308294833637774 1loss/SRqqt2/detection_loss‚1.9258187964558602 .loss/FRt2/detection_loss‚2.413247518918731 % _timestamp‚1.7884436411467335e+09 1loss/SRqqt3/detection_loss‚2.5941481329500675 /loss/FBt4/detection_loss‚1.8899786216872079 epoch‚5ÒwÊ‘˜‚(:† øåÔà—‘É gpu.0.gpu‚91.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚96.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ l6™‚(:« øåÔÊÊþÐ disk.rbd0.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2725.9140625* proc.memory.percent‚16.637659072875977 cpu‚25.295103566923494 proc.cpu.threads‚17 disk.sda.in‚0.5625" disk.dm-0.out‚51380.5107421875 disk.rbd2.out‚520.1904296875' proc.memory.availableMB‚ 1765.109375 network.sent‚ 1.7855908e+07 network.recv‚ 2.217896e+06 disk.sda.out‚ 226.0234375 disk.drbd1777.in‚0$ memory_percent‚89.22662734985352 disk.rbd1.out‚ 3.06640625( disk./.usagePercent‚74.3717767115394 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0% disk./.usageGB‚1108.1932182312012 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1754.234375% disk.nvme0n1.out‚51380.5107421875 disk.sda3.in‚0.5625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚51380.5107421875 disk.sda3.out‚ 226.0234375 disk.dm-0.in‚ 1754.234375 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 1754.234375 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 226.0234375Á†‘š‚(:† øåÔŪòÈ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚37.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rþ¯2›‚(:§ øåÔ×ïþÊ disk.dm-1.out‚ 226.58203125 disk.sda1.in‚0* proc.memory.percent‚16.196823120117188 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd2.out‚520.2685546875 cpu‚25.30597155140216 network.recv‚ 2.224838e+06) disk./.usagePercent‚74.37177543150172 disk.sda3.in‚0.5625 disk.sda3.out‚ 226.58203125! disk.nvme0c0n1.in‚ 1754.234375 disk.drbd1777.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1837.12109375 proc.memory.rssMB‚ 2653.6875 network.sent‚ 1.7953208e+07 disk.rbd3.out‚0# disk.nvme0n1.out‚51381.24609375$ memory_percent‚88.78710269927979 disk.rbd3.in‚0% disk./.usageGB‚1108.1931991577148 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 226.58203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1754.234375 disk.dm-0.in‚ 1754.234375 disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0% disk.nvme0c0n1.out‚51381.24609375 disk.sda.in‚0.5625 disk.rbd0.out‚0 disk.dm-0.out‚51381.24609375 disk.dm-1.in‚0.5625yäV‘œ‚(:† ŸøåÔþÏÉ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚36.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±n=š;‚(:° ŸøåÔãƒáÊ disk.sda.in‚0.5625 disk.rbd3.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚51478.7998046875 disk.sda.out‚ 227.16015625 disk.sda2.out‚0 disk.drbd1777.in‚0 cpu‚25.316736747668784 disk.rbd1.in‚0$ memory_percent‚89.22994136810303 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 1754.234375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚51478.7998046875 disk.drbd1777.out‚0) disk./.usagePercent‚74.37177543150172% disk./.usageGB‚1108.1931991577148 disk.rbd2.out‚521.8037109375 disk.sda1.out‚0 disk.rbd3.in‚0 network.recv‚ 2.229482e+06 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625% disk.nvme0n1.out‚51478.7998046875 disk.rbd0.in‚0 disk.rbd1.out‚ 3.06640625 disk.nvme0n1.in‚ 1754.234375 disk.sda3.out‚ 227.16015625 disk.dm-0.in‚ 1754.234375 disk.dm-1.out‚ 227.16015625) proc.memory.availableMB‚ 1764.56640625" proc.memory.rssMB‚ 2725.9140625* proc.memory.percent‚16.637659072875977 network.sent‚ 1.795897e+07š%¼žž‚(:“ ®øåÔ™´ÉÉ gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚273.925' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.785000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÆ\Ã:Ÿ‚(:¯ ®øåÔâªñÊ disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚89.22884464263916 network.sent‚ 1.7964211e+07 network.recv‚ 2.233217e+06) disk./.usagePercent‚74.37171782980651 disk.sda3.out‚ 227.46484375! disk.nvme0c0n1.in‚ 1754.234375 disk.drbd1777.in‚0 disk.sda.out‚ 227.46484375 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2725.9140625 cpu‚25.327365373095887 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625% disk.nvme0n1.out‚51479.2822265625# disk./.usageGB‚1108.19234085083 disk.rbd1.out‚ 3.06640625 disk.rbd2.out‚523.4560546875 disk.sda2.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚51479.2822265625" disk.dm-0.out‚51479.2822265625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.234375* proc.memory.percent‚16.637659072875977 disk.dm-0.in‚ 1754.234375 disk.dm-1.out‚ 227.46484375 disk.rbd0.out‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1764.74609375åÀÑ«2 - _wandbruntime‚6135  _runtime‚6135ân`#¡òT aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13469411400704 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚13467460682240 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23672651776 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 107255153 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 135406742 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚69605851 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚711336390849536 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚711336239350272 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚823455663147520 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚809986251746816 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚692891929826816 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 _step‚595 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚69605853 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242677869 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚809986227452416 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚823455663147520 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚797019304392704 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚136 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13447075112482 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2649 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975012864 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytº«Dhes.all.freed‚823453688134656 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚138 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3303014400 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1575 $ _timestamp‚1.788443695691037e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚13469411400704 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _runtime‚6135.736072152 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚13467460682240 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2649 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568xˆP}*¢÷T _step‚595 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚69605853 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242677869 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚809986227452416 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚823455663147520 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚797019304392704 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚136 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13447075112482 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2649 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975012864 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚823453688134656 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚138 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3303014400 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1575 $ _timestamp‚1.788443695691037e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚13469411400704 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _runtime‚6135.736072152 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚13467460682240 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2649 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13469411400704 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚13467460682240 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23672651776 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 107255153 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 135406742 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚69605851 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚711336390849536 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚711336239350272 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚823455663147520 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚809986251746816 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚692891929826816 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464Óá«è×2£- _wandbruntime‚6136  _runtime‚6136¨©Y­)¤ûR GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚810536853291874 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚69613783 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚50369206 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚13467934885376 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 135429930 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13469885603840 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2649 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 242695977 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚810060010334720 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚711398847407104 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 135429930 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 135413956 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23672651776 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚823527920925696 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚810059986040320 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 242695977 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 135413956 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚692953912181248 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚810534881158410 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2701149184 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚797089305493504 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23672651776 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚2649 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1575 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚69613781 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 151499264 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13447547798370 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133464 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 _step‚596 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 107266042 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 107266047 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1575 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚50369342 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚711398695907840 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚18444935225856 trainer/global_step‚7253 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 133850624 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚823529895938560 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133464 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 _step‚596 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 107266042 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 107266047 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1575 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚50369342 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚711398695907840 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚18444935225856 trainer/global_step‚7253 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 133850624 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚823529895938560 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28390806913375854 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20605918765068054 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.49711334705352783 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5170066356658936 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5101628303527832 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5223719477653503 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5373627543449402 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03726867213845253 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.18518517911434174 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_5‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3351806402206421 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8127257823944092 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28355854749679565 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_3_of_5‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38845768570899963 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18243825435638428 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9935224652290344 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03520115092396736 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/specificity‚0.9466628432273865 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5678571462631226 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2132936716079712 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007617947179824114 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17198266088962555 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 'particle/f_score‚0.4031170904636383 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4238426685333252 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09857438504695892 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2659098505973816 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1729898452758789 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17920659482479095 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29436030983924866 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16470034420490265 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4162959158420563 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2276342362165451 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1230158805847168 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22318586707115173 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5020752549171448 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9587301015853882 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14222222566604614 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3802827298641205 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6480988264083862 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43486589193344116 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4496996998786926 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26910966634750366 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5151458978652954 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997564971446991 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5170989036560059 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9952720999717712 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9907138347625732 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_3_of_6‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9945067763328552 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5008620023727417 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19752252101898193 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 particle/accuracy_6_of_7‚1 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19061121344566345 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.016421012580394745 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9840491414070129 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7555874586105347 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6034731864929199 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49705687165260315 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02987421490252018 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3863478899002075 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6874999403953552 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5151458978652954 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012089280411601067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5546925067901611 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1131834015250206 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 particle/accuracy_1_of_7‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985976815223694 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5248633027076721 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9856362342834473 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3212963044643402 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4864453673362732 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9763568639755249 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17097002267837524 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.503074586391449 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4335762560367584 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9794888496398926 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9676723480224609 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 particle/accuracy_7_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4044581353664398 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 *jet/accuracy_1_of_4‚0.9215061664581299 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2523438334465027 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5700549483299255 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.989338755607605 particle/accuracy_1_of_6‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.035411980003118515 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5946556925773621 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.28958335518836975 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5406745672225952 ,jet/accuracy_4_of_4‚0.007555942051112652 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9865551590919495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16796502470970154 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9866679906845093 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9817080497741699 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021988593507558107 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1658487766981125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18351665139198303 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3188365697860718 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4341130554676056 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07416921108961105 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 epoch‚5 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5423422455787659 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5504742860794067 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3719136416912079 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9683179259300232 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_3‚0.8732554912567139 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18045978248119354 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01250999141484499 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.007834102027118206 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5313429832458496 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9597700834274292 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5246805548667908 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19430844485759735 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 *jet/accuracy_1_of_5‚0.9700524806976318 particle/accuracy_3_of_4‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4917680621147156 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13782356679439545 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014367816969752312 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11152919381856918 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6874999403953552 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21508055925369263 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_3‚0.244601771235466 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5598304271697998 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20271748304367065 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1340576857328415 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892675280570984 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9949173331260681 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2916666865348816 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 (particle/accuracy‚0.8496404886245728 *particle/sensitivity‚0.325467973947525 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4633333086967468 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29437392950057983 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5061728358268738 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24521459639072418 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_2_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2629629671573639 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5607478022575378 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_6_of_7‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11218202114105225 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.496766060590744 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5650885105133057 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_3_of_3‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884437104780986e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985976815223694 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.17307692766189575 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25898608565330505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5302749872207642 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995068371295929 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 *jet/accuracy_2_of_5‚0.5052867531776428 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16040098667144775 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5010486841201782 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.42549026012420654 jet/accuracy_4_of_7‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.560665488243103 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49715307354927063 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 trainer/global_step‚7253 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4794199764728546 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17192980647087097 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5246398448944092 jet/accuracy_5_of_6‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.48608753085136414 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4638047218322754 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 _runtime‚ 6150.52481532 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975623607635498 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40864449739456177 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2055853009223938 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5909721851348877 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5420830845832825 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016675710678100586 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14025965332984924 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.02376384474337101 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15656036138534546 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.44040149450302124 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.474225252866745 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9854007959365845 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12937156856060028 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34971219301223755 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019936326891183853 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2250000238418579 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.517589271068573 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34449005126953125 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("PUÅëìurity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08048354089260101 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42048609256744385 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9652777314186096 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 particle/accuracy_1_of_1‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.44803935289382935 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19194790720939636 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5383148193359375 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.22003644704818726 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22460433840751648 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9883148670196533 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08593950420618057 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25362929701805115 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5833398103713989 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4834595024585724 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2649233639240265 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4185185432434082 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021722989156842232 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6928374171257019 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4364035129547119 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47826555371284485 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44675692915916443 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1858237385749817 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9854854941368103 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984202980995178 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26959118247032166 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16967836022377014 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03384697809815407 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1526496410369873 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5248990654945374 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09351804107427597 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5843807458877563 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02283320762217045 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07613799721002579 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4149600863456726 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5463876128196716 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5294026732444763 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5165727734565735 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 _step‚597 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18484848737716675 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09109702706336975 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9727822542190552 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_3‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 *loss/val_total_loss‚41.217311865003865 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03115944378077984 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9806229472160339 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *jet/accuracy_2_of_6‚0.5512820482254028§áþaªÖ§ 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25898608565330505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5302749872207642 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995068371295929 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 *jet/accuracy_2_of_5‚0.5052867531776428 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16040098667144775 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5010486841201782 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.42549026012420654 jet/accuracy_4_of_7‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.560665488243103 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.49715307354927063 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 trainer/global_step‚7253 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4794199764728546 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17192980647087097 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5246398448944092 jet/accuracy_5_of_6‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.48608753085136414 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4638047218322754 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 _runtime‚ 6150.52481532 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.975623607635498 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40864449739456177 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2055853009223938 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5909721851348877 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5420830845832825 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016675710678100586 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14025965332984924 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.02376384474337101 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15656036138534546 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.44040149450302124 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.474225252866745 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9854007959365845 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12937156856060028 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34971219301223755 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019936326891183853 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2250000238418579 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.517589271068573 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34449005126953125 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08048354089260101 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42048609256744385 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9652777314186096 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 particle/accuracy_1_of_1‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.44803935289382935 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19194790720939636 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5383148193359375 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.22003644704818726 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4236111044883728 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22460433840751648 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9883148670196533 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08593950420618057 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25362929701805115 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5833398103713989 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4834595024585724 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2649233639240265 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4185185432434082 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021722989156842232 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6928374171257019 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4364035129547119 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47826555371284485 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44675692915916443 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1858237385749817 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9854854941368103 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984202980995178 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26959118247032166 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16967836022377014 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03384697809815407 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1526496410369873 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5248990654945374 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09351804107427597 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5843807458877563 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02283320762217045 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07613799721002579 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4149600863456726 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5463876128196716 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5294026732444763 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5165727734565735 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 _step‚597 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18484848737716675 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09109702706336975 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9727822542190552 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_3‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 *loss/val_total_loss‚41.217311865003865 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03115944378077984 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9806229472160339 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *jet/accuracy_2_of_6‚0.5512820482254028 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802090167999 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.984307050704956 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15749993920326233 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17034713923931122 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3827519118785858 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06666666269302368 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16979454457759857 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2322530746459961 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17920659482479095 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20559269189834595 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9847166538238525 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5666038990020752 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.28390806913375854 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20605918765068054 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.49711334705352783 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5170066356658936 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5101628303527832 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5223719477653503 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5373627543449402 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03726867213845253 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.18518517911434174 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_5‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3351806402206421 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8127257823944092 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28355854749679565 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_3_of_5‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38845768570899963 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18243825435638428 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9935224652290344 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03520115092396736 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/specificity‚0.9466628432273865 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5678571462631226 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2132936716079712 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007617947179824114 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17198266088962555 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 'particle/f_score‚0.4031170904636383 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4238426685333252 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09857438504695892 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2659098505973816 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1729898452758789 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17920659482479095 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29436030983924866 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16470034420490265 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4162959158420563 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2276342362165451 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1230158805847168 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22318586707115173 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5020752549171448 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9587301015853882 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14222222566604614 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3802827298641205 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6480988264083862 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43486589193344116 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4496996998786926 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26910966634750366 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5151458978652954 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997564971446991 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5170989036560059 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9952720999717712 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9907138347625732 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_3_of_6‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9945067763328552 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5008620023727417 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19752252101898193 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 particle/accuracy_6_of_7‚1 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19061121344566345 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.016421012580394745 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9840491414070129 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7555874586105347 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.000992063549347221HO¡K×29 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6034731864929199 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49705687165260315 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02987421490252018 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3863478899002075 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6874999403953552 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5151458978652954 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012089280411601067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5546925067901611 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1131834015250206 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 particle/accuracy_1_of_7‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985976815223694 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5248633027076721 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9856362342834473 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3212963044643402 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4864453673362732 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9763568639755249 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17097002267837524 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.503074586391449 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4335762560367584 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9794888496398926 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9676723480224609 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 particle/accuracy_7_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4044581353664398 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 *jet/accuracy_1_of_4‚0.9215061664581299 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2523438334465027 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5700549483299255 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.989338755607605 particle/accuracy_1_of_6‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.035411980003118515 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5347222089767456 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5946556925773621 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.28958335518836975 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5406745672225952 ,jet/accuracy_4_of_4‚0.007555942051112652 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9865551590919495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16796502470970154 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9866679906845093 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9817080497741699 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021988593507558107 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1658487766981125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18351665139198303 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3188365697860718 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4341130554676056 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07416921108961105 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 epoch‚5 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5423422455787659 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5504742860794067 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3719136416912079 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9683179259300232 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_3‚0.8732554912567139 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18045978248119354 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01250999141484499 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.007834102027118206 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5313429832458496 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9597700834274292 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5246805548667908 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19430844485759735 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 *jet/accuracy_1_of_5‚0.9700524806976318 particle/accuracy_3_of_4‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4917680621147156 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13782356679439545 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014367816969752312 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11152919381856918 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6874999403953552 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21508055925369263 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 )jet/accuracy_2_of_3‚0.244601771235466 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5598304271697998 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20271748304367065 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1340576857328415 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892675280570984 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9949173331260681 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2916666865348816 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 (particle/accuracy‚0.8496404886245728 *particle/sensitivity‚0.325467973947525 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4633333086967468 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29437392950057983 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5061728358268738 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24521459639072418 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_2_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2629629671573639 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5607478022575378 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_6_of_7‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11218202114105225 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.496766060590744 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5650885105133057 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 particle/accuracy_3_of_3‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884437104780986e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985976815223694 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.17307692766189575 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297Õé÷ÍŸ”«‚(:‰ ÌøåÔÓüÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚235.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a°ú:¬‚(:¯ ÌøåÔÐçÔÊ' disk.nvme0c0n1.out‚56291.2001953125 disk.dm-1.out‚ 228.5546875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0) proc.memory.percent‚16.63792133331299 disk.sda2.in‚0 network.sent‚ 1.8079459e+07$ disk./.usageGB‚1108.193286895752 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd1.out‚ 3.06640625) disk./.usagePercent‚74.37178131967501" disk.nvme0c0n1.in‚ 1754.8203125 disk.dm-0.in‚ 1754.8203125 disk.drbd1777.in‚0) proc.memory.availableMB‚ 3748.11328125# proc.memory.rssMB‚ 2725.95703125 disk.rbd0.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚56291.2001953125 disk.sda3.out‚ 228.5546875 disk.rbd2.in‚0 disk.sda1.out‚0# memory_percent‚77.1233320236206 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1754.8203125 disk.sda3.in‚0.5625 disk.rbd2.out‚525.1357421875 disk.sda.out‚ 228.5546875 disk.rbd3.in‚0" disk.dm-0.out‚56291.2001953125 disk.dm-1.in‚0.5625 cpu‚25.338418594813376 network.recv‚ 2.244172e+06 disk.rbd0.out‚0ÍÐë”­‚(:‰ ÛøåÔëåèÈ gpu.0.gpu‚87.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚129.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß;O<®‚(:± ÛøåÔÇÊèË disk.sda3.in‚0.5625) disk./.usagePercent‚74.37178515978803" disk.nvme0c0n1.in‚ 1754.8203125 disk.dm-1.out‚ 229.19140625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚57060.81103515625$ proc.memory.availableMB‚3748.125 cpu‚25.337711884099303 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.06640625 disk.rbd2.out‚525.1591796875 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚57060.81103515625 network.sent‚ 1.8204966e+07# proc.memory.rssMB‚ 2725.95703125) proc.memory.percent‚16.63792133331299 network.recv‚ 2.251609e+06 disk.dm-0.in‚ 1754.8203125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda.out‚ 229.19140625 disk.sda2.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚57060.81103515625 disk.nvme0n1.in‚ 1754.8203125$ memory_percent‚77.12326049804688 disk.sda3.out‚ 229.19140625$ disk./.usageGB‚1108.193344116211ÔZÓ¯‚(:… êøåÔ‰œ…É gpu.0.gpu‚85.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.2' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.64 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™R=°‚(:² êøåÔÜÂ’Ë( disk./.usagePercent‚74.3717977041572 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1754.8203125 cpu‚25.337023803088094 network.sent‚ 1.8210176e+07# proc.memory.rssMB‚ 2725.95703125 network.recv‚ 2.255336e+06 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625' disk.nvme0c0n1.out‚57158.6083984375$ memory_percent‚77.12323665618896" disk.dm-0.out‚57158.6083984375 disk.sda1.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 229.49609375 disk.rbd2.out‚526.7490234375 disk.sda1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.63792133331299$ disk./.usageGB‚1108.193531036377 disk.dm-0.in‚ 1754.8203125 disk.sda.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 229.49609375 disk.rbd3.in‚0) proc.memory.availableMB‚ 3748.12890625 proc.cpu.threads‚17 disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0% disk.nvme0n1.out‚57158.6083984375 disk.sda3.out‚ 229.49609375" disk.nvme0c0n1.in‚ 1754.8203125zÏO±2±- _wandbruntime‚6196  _runtime‚6196ŽÅ¡àƲÀ & lr-AdamW/pg2‚0.0001576352498872853 trainer/global_step‚7299 % _timestamp‚1.7884437563042555e+09 _runtime‚ 6196.34891124 _step‚598 & lr-AdamW/pg1‚0.00015763524988728534ƒÛ˳Š_runtime‚ 6196.34891124 _step‚598 & lr-AdamW/pg1‚0.0001576352498872853 & lr-AdamW/pg2‚0.0001576352498872853 trainer/global_step‚7299 % _timestamp‚1.7884437563042555e+09ÖæVän2´- _wandbruntime‚6196  _runtime‚6196Þ<¹à(µŠP KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13555923064832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50726744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70130683 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚697917160755200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚829432051289088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚829434677890048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70130686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚802784672646144 _runtime‚6196.349821505 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚697917132620800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13555923064832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 % _timestamp‚1.7884437563049588e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚815876154615808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚816320701828460 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚815876128224256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136330565 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18565561395200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13558523274240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716482722150400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚829432051289088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120857890 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚815876128224256 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716482645522432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚816318078115824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚802784647218176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120857427 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13536029182316 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108048289 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13558523274240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 trainer/global_step‚7299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1üH{7^¶P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚829432051289088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120857890 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚815876128224256 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716482645522432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚816318078115824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚802784647218176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120857427 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13536029182316 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108048289 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13558523274240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 trainer/global_step‚7299 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13555923064832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50726744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70130683 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚697917160755200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚829432051289088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚829434677890048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70130686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚802784672646144 _runtime‚6196.349821505 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚697917132620800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13555923064832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 % _timestamp‚1.7884437563049588e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚815876154615808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚816320701828460 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚815876128224256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136330565 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18565561395200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13558523274240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716482722150400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649×B*62·- _wandbruntime‚6197  _runtime‚6197ðàñ'¸“N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13557755393536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136348799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716577637435392 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716577560820736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚815984094321152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚815984067929600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚802890988392448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788443757044139e+09 _runtime‚6197.089317774 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244411318 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13535260240316 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120873387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13560355616256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18568088626688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚816428872345608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚802891013820416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108062519 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120873831 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50733949 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136368859 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚829544449937408 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚815984094321152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚815984067929600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13537858525192?œ¤''¹˜N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13557755393536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136348799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716577637435392 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716577560820736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚815984094321152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚815984067929600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚802890988392448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788443757044139e+09 _runtime‚6197.089317774 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244411318 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13535260240316 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120873387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13560355616256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18568088626688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚816428872345608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚802891013820416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108062519 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120873831 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50733949 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136368859 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚829544449937408 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚815984094321152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚815984067929600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13537858525192ØÀÞQH2º- _wandbruntime‚6197  _runtime‚6197c©?µZ»Ô 1loss/FBt1/assignment_loss‚0.14671222580464865 1loss/FBt3/assignment_loss‚0.24232568465231452 /loss/FRt3/detection_loss‚2.7532153294029422 &loss/total_loss‚40.770885079650355 .loss/FBt2/detection_loss‚2.507852740585804 0loss/FRt2/assignment_loss‚2.5373291955752806 2loss/SRqqt1/assignment_loss‚1.1186268531426322 2loss/SRqqt2/assignment_loss‚1.0769226523382323 2loss/SRqqt4/assignment_loss‚1.2714274287299843 /loss/FRt4/detection_loss‚2.6221427112012297 .loss/FBt1/detection_loss‚0.520453193272395 0loss/FRt4/assignment_loss‚2.5438405597532117 2loss/SRqqt3/assignment_loss‚1.6083739689654775 /loss/FRt1/detection_loss‚2.2736968511866555 1loss/SRqqt2/detection_loss‚2.0696662204606193 /loss/FBt4/detection_loss‚2.5277635753154755 epoch‚6 /loss/FRt3/assignment_loss‚3.393134150011786 0loss/FBt4/assignment_loss‚0.3899919195100665 % _timestamp‚1.7884437570461774e+09 .loss/FBt3/detection_loss‚2.405346430838108 /loss/FRt1/assignment_loss‚1.059056356146529 1loss/SRqqt1/detection_loss‚0.8993058339334451 _runtime‚6197.090740764 0loss/FBt2/assignment_loss‚0.4977729136007838 -loss/FRt2/detection_loss‚2.40220153873617 1loss/SRqqt4/detection_loss‚1.7258206455379117 trainer/global_step‚7299 _step‚601 0loss/SRqqt3/detection_loss‚2.177906100948652çùhj_¼Ù _runtime‚6197.090740764 0loss/FBt2/assignment_loss‚0.4977729136007838 -loss/FRt2/detection_loss‚2.40220153873617 1loss/SRqqt4/detection_loss‚1.7258206455379117 trainer/global_step‚7299 _step‚601 0loss/SRqqt3/detection_loss‚2.177906100948652 1loss/FBt1/assignment_loss‚0.14671222580464865 1loss/FBt3/assignment_loss‚0.24232568465231452 /loss/FRt3/detection_loss‚2.7532153294029422 &loss/total_loss‚40.770885079650355 .loss/FBt2/detection_loss‚2.507852740585804 0loss/FRt2/assignment_loss‚2.5373291955752806 2loss/SRqqt1/assignment_loss‚1.1186268531426322 2loss/SRqqt2/assignment_loss‚1.0769226523382323 2loss/SRqqt4/assignment_loss‚1.2714274287299843 /loss/FRt4/detection_loss‚2.6221427112012297 .loss/FBt1/detection_loss‚0.520453193272395 0loss/FRt4/assignment_loss‚2.5438405597532117 2loss/SRqqt3/assignment_loss‚1.6083739689654775 /loss/FRt1/detection_loss‚2.2736968511866555 1loss/SRqqt2/detection_loss‚2.0696662204606193 /loss/FBt4/detection_loss‚2.5277635753154755 epoch‚6 /loss/FRt3/assignment_loss‚3.393134150011786 0loss/FBt4/assignment_loss‚0.3899919195100665 % _timestamp‚1.7884437570461774e+09 .loss/FBt3/detection_loss‚2.405346430838108 /loss/FRt1/assignment_loss‚1.059056356146529 1loss/SRqqt1/detection_loss‚0.8993058339334451ÙÄ;áž½‚(:“ ùøåÔ”ç¬É gpu.0.gpu‚58.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.367999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dz5¾‚(:ª ùøåÔª„‡Ì' disk.nvme0c0n1.out‚57158.9326171875 network.sent‚ 1.8216175e+07) disk./.usagePercent‚74.37178823187843 disk.rbd2.out‚526.7763671875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 230.09375 disk.dm-1.out‚ 230.09375" disk.nvme0c0n1.in‚ 1754.8203125% disk./.usageGB‚1108.1933898925781 disk.rbd1.out‚ 3.06640625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1754.8203125 cpu‚25.336285826475486 disk.rbd1.in‚0 disk.sda.in‚0.5625% disk.nvme0n1.out‚57158.9326171875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0) proc.memory.percent‚16.63792133331299 disk.dm-0.in‚ 1754.8203125" disk.dm-0.out‚57158.9326171875 network.recv‚ 2.259541e+06 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚77.12187767028809( proc.memory.availableMB‚ 3748.3515625 disk.rbd0.out‚0 disk.sda3.in‚0.5625# proc.memory.rssMB‚ 2725.95703125 disk.sda.out‚ 230.09375 disk.sda1.in‚0 disk.dm-1.in‚0.5625;Ì“¿‚(:ˆ ˆùåÔ¯––É gpu.0.gpu‚67.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚256.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.247 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…£ÄºAÀ‚(:¶ ˆùåÔ¥ÆøË disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0# disk.dm-0.out‚57256.55615234375% disk./.usageGB‚1108.1933326721191 disk.sda.in‚0.5625( disk.nvme0c0n1.out‚57256.55615234375 network.recv‚ 2.266646e+06) disk./.usagePercent‚74.37178439176542 disk.rbd1.in‚0 disk.sda.out‚ 230.51953125 disk.sda2.out‚0) proc.memory.percent‚16.63792133331299 network.sent‚ 1.8325582e+07 disk.rbd3.in‚0 disk.sda3.out‚ 230.51953125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1754.8203125# proc.memory.rssMB‚ 2725.95703125 disk.sda2.in‚0 disk.nvme0n1.in‚ 1754.8203125 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.8203125 disk.dm-1.out‚ 230.51953125 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚77.12359428405762 cpu‚25.335599840893703 proc.cpu.threads‚17& disk.nvme0n1.out‚57256.55615234375 disk.rbd2.out‚528.4912109375 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625( proc.memory.availableMB‚ 3748.0703125c9Õú2Á- _wandbruntime‚6235  _runtime‚6235íUýjÇÂÁ _step‚602 & lr-AdamW/pg1‚0.0001548910426947381 & lr-AdamW/pg2‚0.0001548910426947381 trainer/global_step‚7349 % _timestamp‚1.7884437954091725e+09 _runtime‚6235.453825581´„$ÌÃÆ _runtime‚6235.453825581 _step‚602 & lr-AdamW/pg1‚0.0001548910426947381 & lr-AdamW/pg2‚0.0001548910426947381 trainer/global_step‚7349 % _timestamp‚1.7884437954091725e+09Úÿ ~%2Ä- _wandbruntime‚6235  _runtime‚6235Ú®ÏjçÅŠP JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721228225417216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821726617554032 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70590483 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121654353 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246025740 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121653871 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13627509906924 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51063867 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834923294006272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108760083 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _runtime‚6235.454787944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚821273113491456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13647553913856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834920667405312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721228302045184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚821273139883008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702536535296000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 trainer/global_step‚7349 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13650154123264 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884437954099011e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137265651 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚834923294006272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137245617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ãOÕ0(ÆP _step‚603 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚821273113491456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70590486 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702536563430400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚834920667405312 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108760089 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108760083 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834923294006272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108760083 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _runtime‚6235.454787944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚821273113491456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13647553913856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834920667405312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721228302045184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚821273139883008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702536535296000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 trainer/global_step‚7349 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13650154123264 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 % _timestamp‚1.7884437954099011e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137265651 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚834923294006272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137245617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721228225417216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821726617554032 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70590483 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121654353 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246025740 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121653871 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13627509906924 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51063867 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51070115 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18694215800320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13626740964924 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821834788070972 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137283911 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121670253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51070571 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚821381053196800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13649386242560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821837411783816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚808208072534016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137283911 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884437961447928e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121669794 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108774319 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137263851 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13651986465280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13629339249800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚821381053196800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚821381079588352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702628923349504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚821381079588352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702628923349504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51070115 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18694215800320 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13626740964924 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821834788070972 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137283911 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121670253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51070571 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceSt¯^ì6ÌatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚821381053196800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13649386242560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821837411783816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚808208072534016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137283911 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884437961447928e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121669794 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108774319 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137263851 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13651986465280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13629339249800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚821381053196800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109471883 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840409283521536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13739184762880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13739184762880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122450528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884438343494806e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247632552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138180703 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813418790073344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚826670098758656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138160669 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827137780704876 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109471883 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚725974284804096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 trainer/global_step‚7399 _step‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109471889 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚827135156992240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18818391132160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18818346832896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840411910122496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13741784972288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚707155966105600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840411910122496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840409283521536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚826670125150208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51400245í¯l"IØP UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚827135156992240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18818391132160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18818346832896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840411910122496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13741784972288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚707155966105600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840411910122496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840409283521536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚826670125150208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51400245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚725974357237760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚826670125150208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13741784972288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71050286 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51400662 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13718990631532 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109471889 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122450948 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚707155937971200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109471883 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840409283521536 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XEDeviceStatsMonitor.on_traiâ“î`Ì n_batch_start/active_bytes.small_pool.freed‚13739184762880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13739184762880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122450528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884438343494806e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247632552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138180703 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813418790073344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚826670098758656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138160669 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827137780704876 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109471883 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚725974284804096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 trainer/global_step‚7399 _step‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109471889 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575ß¿Iµ2Ù- _wandbruntime‚6275  _runtime‚6275 ÷B'Ú“N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13741017091584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚726069277415936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826778038464000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138198963 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122466390 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚827245951222024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚813525105819648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109486125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826778064855552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109486119 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _runtime‚6275.129410685 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18820881068032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71059479 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚707248326024704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840519055555584 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109486119 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247665022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51406911 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138178903 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138178903 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13720819974408 $ _timestamp‚1.788443835084261e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13741017091584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71059482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 trainer/global_step‚7399 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109486125 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826778064855552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13743617314304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚827243327509180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚813525131247616 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109486119 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247665022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51406911 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138178903 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138178903 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13720819974408 $ _timestamp‚1.788443835084261e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13741017091584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71059482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 trainer/global_step‚7399 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109486125 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826778064855552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13743617314304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚827243327509180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚813525131247616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13741017091584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚726069277415936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826778038464000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138198963 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122466390 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚827245951222024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚813525105819648 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109486125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826778064855552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109486119 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _runtime‚6275.129410685 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18820881068032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71059479 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚707248326024704à#гj2Ü- _wandbruntime‚6275  _runtime‚6275 YÝÓ -loss/FBt4/detection_loss‚3.25392846763134 $loss/total_loss‚41.2591544527174 epoch‚6 % _timestamp‚1.7884438350858839e+09 _runtime‚ 6275.13044309 /loss/FBt2/detection_loss‚1.9551288052038713 0loss/FBt4/assignment_loss‚0.2123363777063787 /loss/FRt2/detection_loss‚2.3539870745958167 2loss/SRqqt1/assignment_loss‚1.2399072907592785 trainer/global_step‚7399 1loss/FBt3/assignment_loss‚0.01084034207451623 1loss/SRqqt2/detection_loss‚1.9263140878012015 _step‚609 0loss/FRt2/assignment_loss‚2.9912138822230885 2loss/SRqqt3/assignment_loss‚1.5742360209265063 1loss/FBt1/assignment_loss‚0.08460101642805189 1loss/FBt2/assignment_loss‚0.23990398112271744 1loss/SRqqt1/detection_loss‚0.8896109726159803 /loss/FRt4/assignment_loss‚3.234810857545762 .loss/FRt4/detection_loss‚2.577416644919486 0loss/FBt1/detection_loss‚0.47558706294054565 0loss/FRt1/assignment_loss‚1.3309418521821499 .loss/FRt3/detection_loss‚2.349354546492072 0loss/SRqqt3/detection_loss‚2.328280564277403 2loss/SRqqt4/assignment_loss‚1.1209787066937202 .loss/FRt1/detection_loss‚1.836185229709372 /loss/FBt3/detection_loss‚3.2138825058937073 /loss/FRt3/assignment_loss‚3.021060135629442 2loss/SRqqt2/assignment_loss‚1.1397148838569953 1loss/SRqqt4/detection_loss‚1.8989331434879984GÙ]…^ÞØ 2loss/SRqqt4/assignment_loss‚1.1209787066937202 .loss/FRt1/detection_loss‚1.836185229709372 /loss/FBt3/detection_loss‚3.2138825058937073 /loss/FRt3/assignment_loss‚3.021060135629442 2loss/SRqqt2/assignment_loss‚1.1397148838569953 1loss/SRqqt4/detection_loss‚1.8989331434879984 -loss/FBt4/detection_loss‚3.25392846763134 $loss/total_loss‚41.2591544527174 epoch‚6 % _timestamp‚1.7884438350858839e+09 _runtime‚ 6275.13044309 /loss/FBt2/detection_loss‚1.9551288052038713 0loss/FBt4/assignment_loss‚0.2123363777063787 /loss/FRt2/detection_loss‚2.3539870745958167 2loss/SRqqt1/assignment_loss‚1.2399072907592785 trainer/global_step‚7399 1loss/FBt3/assignment_loss‚0.01084034207451623 1loss/SRqqt2/detection_loss‚1.9263140878012015 _step‚609 0loss/FRt2/assignment_loss‚2.9912138822230885 2loss/SRqqt3/assignment_loss‚1.5742360209265063 1loss/FBt1/assignment_loss‚0.08460101642805189 1loss/FBt2/assignment_loss‚0.23990398112271744 1loss/SRqqt1/detection_loss‚0.8896109726159803 /loss/FRt4/assignment_loss‚3.234810857545762 .loss/FRt4/detection_loss‚2.577416644919486 0loss/FBt1/detection_loss‚0.47558706294054565 0loss/FRt1/assignment_loss‚1.3309418521821499 .loss/FRt3/detection_loss‚2.349354546492072 0loss/SRqqt3/detection_loss‚2.328280564277403áèM“ß‚(:ˆ ÄùåÔèûšÉ gpu.0.gpu‚70.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.857 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[r‰Aà‚(:¶ ÄùåÔŸ—Ë disk.rbd3.in‚0& disk.nvme0n1.out‚57355.87744140625 disk.sda.in‚0.5625" disk.nvme0c0n1.in‚ 1754.8203125 disk.dm-0.in‚ 1754.8203125 disk.dm-1.in‚0.5625 cpu‚25.332891328858434 proc.cpu.threads‚17 disk.sda.out‚ 232.58984375 network.sent‚ 1.845247e+07) disk./.usagePercent‚74.37178464777296 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚57355.87744140625) proc.memory.percent‚16.63792133331299 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚0.5625 disk.dm-1.out‚ 232.58984375 disk.rbd3.out‚0% disk./.usageGB‚1108.1933364868164 disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0# proc.memory.rssMB‚ 2725.95703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 1754.8203125 disk.sda3.out‚ 232.58984375$ memory_percent‚77.12671756744385) proc.memory.availableMB‚ 3747.55859375 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚57355.87744140625 disk.rbd2.out‚531.7841796875 network.recv‚ 2.285872e+06 disk.rbd1.in‚0;Ë~µŸá‚(:” ÓùåÔšáÓÈ gpu.0.gpu‚94.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚259.478' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.895599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†–û9â‚(:® ÓùåÔ§áÕË disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.8562269e+07) disk./.usagePercent‚74.37175495089899 disk.rbd0.out‚0 disk.dm-0.in‚ 1754.8203125 disk.dm-1.out‚ 233.109375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 1754.8203125 disk.sda.out‚ 233.109375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.in‚0.5625 disk.sda3.out‚ 233.109375$ memory_percent‚77.12829113006592 network.recv‚ 2.292883e+06" disk.dm-0.out‚57456.7861328125 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 3747.30078125 cpu‚25.332177158627665% disk.nvme0n1.out‚57456.7861328125 disk.rbd1.out‚ 3.06640625 disk.nvme0n1.in‚ 1754.8203125) proc.memory.percent‚16.63792133331299 disk.sda1.in‚0# proc.memory.rssMB‚ 2725.95703125 disk.drbd1777.out‚0% disk./.usageGB‚1108.1928939819336 disk.rbd0.in‚0' disk.nvme0c0n1.out‚57456.7861328125 disk.rbd2.out‚533.3193359375 disk.sda3.in‚0.5625 proc.cpu.threads‚17WŠBx2ã- _wandbruntime‚6313  _runtime‚6313éòº¼Éäà trainer/global_step‚7449 % _timestamp‚1.7884438734262574e+09 _runtime‚6313.470916568 _step‚610 ' lr-AdamW/pg1‚0.00014942749736150833 ' lr-AdamW/pg2‚0.00014942749736150833ÔĦÎåÈ ' lr-AdamW/pg1‚0.00014942749736150833 ' lr-AdamW/pg2‚0.00014942749736150833 trainer/global_step‚7449 % _timestamp‚1.7884438734262574e+09 _runtime‚6313.470916568 _step‚610âËé§2æ- _wandbruntime‚6313  _runtime‚6313jôæ¾Ë çŠP MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139075721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71510083 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110183689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚818735848786944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚832067084025856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13830815611904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚845897899637760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139095755 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚845900526238720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139095755 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711775368780800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_starʉ«´Et/allocated_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139075721 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚832543696430448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13833415821312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚818735823358976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13830815611904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123248965 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚845900526238720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚611 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110183683 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13810471356140 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚845897899637760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123248507 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18945034599424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71510086 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711775340646400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚730720447679488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884438734270122e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚832546320143084 .‚?(èP JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71510083 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110183689 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚818735848786944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚832067084025856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13830815611904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚845897899637760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139095755 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚845900526238720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139095755 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711775368780800 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139075721 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚832543696430448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13833415821312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚818735823358976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13830815611904 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123248965 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚845900526238720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚611 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110183683 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13810471356140 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚845897899637760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123248507 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18945034599424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71510086 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711775340646400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚730720447679488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884438734270122e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚832546320143084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139075721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123264493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚832654490660232 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139093955 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚711867756834304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71519282 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13835248163328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110197925 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚730815365834240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13809702414140 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139114015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚832175023731200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13832647940608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71519279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚832175023731200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51745625 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 trainer/global_step‚7449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123264907 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6314.204598596 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚818842164533248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚711867728699904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚818842189961216 $ _timestamp‚1.788443874159287e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672NÇÖà8ë˜N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51745214 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚832651866947388 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚846007671671808 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110197925 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13835248163328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110197919 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846010298286080 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139093955 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13832647940608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110197919 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13812300699016 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249311940 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123264493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚832654490660232 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139093955 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚711867756834304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71519282 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13835248163328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110197925 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚730815365834240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13809702414140 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139114015 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚832175023731200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13832647940608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71519279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚832175023731200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51745625 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 trainer/global_step‚7449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123264907 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6314.204598596 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚818842164533248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚711867728699904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚818842189961216 $ _timestamp‚1.788443874159287e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚851389142354944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139990773 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚837952235868656 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 trainer/global_step‚7499 _runtime‚6352.195387305 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140010807 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚824052907500544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110895489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚837464095684608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13925046670336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚851386515753984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788443912150519e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚851386515753984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13899353796080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚837464095684608 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19071559568896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13901952080748 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139990773 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚735466254396928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250886256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13922446460928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52074114 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110895483 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚824052882072576 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575k¾eSH'úŽP `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚824052907500544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110895489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚837464095684608 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13925046670336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚851386515753984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788443912150519e+09 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚851386515753984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13899353796080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚837464095684608 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19071559568896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13901952080748 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139990773 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚735466254396928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250886256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13922446460928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52074114 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110895483 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚824052882072576 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71969883 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚837464069293056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚735466331024896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19071511075328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚837464069293056 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124043997 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚851389142354944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139990773 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚837952235868656 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatj 4¢ÌsMonitor.on_train_batch_start/segment.all.peak‚2649 trainer/global_step‚7499 _runtime‚6352.195387305 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140010807ç[ñï%2û- _wandbruntime‚6352  _runtime‚6352ØL'ü”N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250918726 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 trainer/global_step‚7499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110909725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚824159223246848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716487159509504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13924278789632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124060347 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚837572035389952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52081265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13926879012352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71979082 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19074027893760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13903781423624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚735561233786368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚824159248674816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚838063030098440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124059909 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚837572008998400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚837572008998400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19074074276864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚716487159509504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13924278789632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124060347 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚837572035389952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52081265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13926879012352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71979082 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19074027893760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13903781423624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚735561233786368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚824159248674816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚838063030098440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124059909 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚837572008998400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚837572008998400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19074074276864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844èÇhDú2þ- _wandbruntime‚6352  _runtime‚6352MmS\ÿÖ 1loss/FBt1/assignment_loss‚0.14534500612031087 1loss/SRqqt2/detection_loss‚2.0368619193422033 .loss/FBt4/detection_loss‚2.981840640306473 _runtime‚6352.930579134 /loss/FRt2/assignment_loss‚2.745441371574998 _step‚617 1loss/SRqqt4/detection_loss‚1.7448809019945286 epoch‚6 trainer/global_step‚7499 /loss/FRt4/assignment_loss‚2.668623125553131 2loss/SRqqt3/assignment_loss‚1.3911129798119266 0loss/FBt3/assignment_loss‚0.6970733223882105 /loss/FBt3/detection_loss‚3.0166230542319163 %loss/total_loss‚41.52170600317949 /loss/FRt3/assignment_loss‚3.063391775656969 /loss/FRt2/detection_loss‚2.8747537583112717 % _timestamp‚1.7884439128860102e+09 1loss/SRqqt1/assignment_loss‚0.768992636386039 2loss/SRqqt4/assignment_loss‚1.2928640151189432 /loss/FRt4/detection_loss‚2.3704275391995906 /loss/FBt1/detection_loss‚0.5379952026525205 .loss/FRt1/assignment_loss‚1.79052611914548 2loss/SRqqt2/assignment_loss‚1.0584730421926112 1loss/FBt2/assignment_loss‚0.06279368514660746 2loss/FBt4/assignment_loss‚0.006051792064681649 /loss/FRt1/detection_loss‚1.6832504514604807 /loss/FBt2/detection_loss‚2.9351683020591737 /loss/FRt3/detection_loss‚2.2910256647558596 1loss/SRqqt1/detection_loss‚0.8212857811578682 1loss/SRqqt3/detection_loss‚2.53690391654769568¬Ia€Û /loss/FRt3/detection_loss‚2.2910256647558596 1loss/SRqqt1/detection_loss‚0.8212857811578682 1loss/SRqqt3/detection_loss‚2.5369039165476956 1loss/FBt1/assignment_loss‚0.14534500612031087 1loss/SRqqt2/detection_loss‚2.0368619193422033 .loss/FBt4/detection_loss‚2.981840640306473 _runtime‚6352.930579134 /loss/FRt2/assignment_loss‚2.745441371574998 _step‚617 1loss/SRqqt4/detection_loss‚1.7448809019945286 epoch‚6 trainer/global_step‚7499 /loss/FRt4/assignment_loss‚2.668623125553131 2loss/SRqqt3/assignment_loss‚1.3911129798119266 0loss/FBt3/assignment_loss‚0.6970733223882105 /loss/FBt3/detection_loss‚3.0166230542319163 %loss/total_loss‚41.52170600317949 /loss/FRt3/assignment_loss‚3.063391775656969 /loss/FRt2/detection_loss‚2.8747537583112717 % _timestamp‚1.7884439128860102e+09 1loss/SRqqt1/assignment_loss‚0.768992636386039 2loss/SRqqt4/assignment_loss‚1.2928640151189432 /loss/FRt4/detection_loss‚2.3704275391995906 /loss/FBt1/detection_loss‚0.5379952026525205 .loss/FRt1/assignment_loss‚1.79052611914548 2loss/SRqqt2/assignment_loss‚1.0584730421926112 1loss/FBt2/assignment_loss‚0.06279368514660746 2loss/FBt4/assignment_loss‚0.006051792064681649 /loss/FRt1/detection_loss‚1.6832504514604807 /loss/FBt2/detection_loss‚2.9351683020591737écÐ+ž‚(:“ úåÔéߘÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚255.414' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.08279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½{Ç?=‚‚(:² úåÔ®õ”Ë# disk.nvme0c0n1.in‚ 1754.82421875' disk.nvme0c0n1.out‚57556.0400390625 disk.dm-1.out‚ 234.9921875) proc.memory.percent‚16.63792133331299 disk.sda.out‚ 234.9921875 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.dm-0.in‚ 1754.82421875" disk.dm-0.out‚57556.0400390625 disk.rbd2.out‚536.6748046875 disk.rbd3.out‚0 disk.sda3.out‚ 234.9921875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0.5625( proc.memory.availableMB‚ 3747.6484375 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0$ memory_percent‚77.12616920471191# proc.memory.rssMB‚ 2725.95703125) disk./.usagePercent‚74.37178464777296 disk.rbd1.out‚ 3.06640625 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚ 2.311488e+06% disk./.usageGB‚1108.1933364868164 disk.sda.in‚0.5625% disk.nvme0n1.out‚57556.0400390625 disk.dm-1.in‚0.5625 cpu‚25.32945209244567 network.sent‚ 1.8688188e+07 disk.sda2.in‚00¦Õ‚(:‰ žúåÔÏ·úÈ gpu.0.gpu‚82.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚102.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ñŽB„‚(:· žúåÔꥎË disk.rbd2.out‚538.3349609375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1754.82421875# disk.dm-0.out‚57653.96923828125 disk.rbd1.out‚ 3.06640625 disk.rbd0.in‚0 cpu‚25.3287977341213 disk.dm-0.in‚ 1754.82421875 network.sent‚ 1.8796937e+07 disk.sda.out‚ 235.29296875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 235.29296875 disk.drbd1777.in‚0 network.recv‚ 2.317673e+06) disk./.usagePercent‚74.37178183169007 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚57653.96923828125 disk.dm-1.out‚ 235.29296875) proc.memory.percent‚16.02635383605957 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.drbd1777.out‚0) proc.memory.availableMB‚ 3844.83984375" proc.memory.rssMB‚ 2625.7578125 disk.dm-1.in‚0.5625% disk./.usageGB‚1108.1932945251465& disk.nvme0n1.out‚57653.96923828125$ memory_percent‚76.53295993804932 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1754.82421875úÄÌ>ž…‚(:“ ­úåÔùÇþÈ gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚91.826' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.365199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ÓŒ92†‚(:§ ­úåÔºèÝË network.recv‚ 2.321971e+06 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.37174189451474 disk.rbd3.out‚0$ memory_percent‚89.13776874542236 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 1779.66796875 disk.drbd1777.in‚0 disk.dm-0.in‚ 1754.82421875$ disk./.usageGB‚1108.192699432373 disk.drbd1777.out‚0 disk.dm-1.in‚0.5625# proc.memory.rssMB‚ 2628.43359375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd2.out‚538.3583984375 disk.sda.in‚0.5625 disk.sda.out‚235.875 disk.sda2.out‚0% disk.nvme0n1.out‚57654.2783203125 disk.rbd0.in‚0* proc.memory.percent‚16.042685508728027 network.sent‚ 1.8802879e+07 disk.rbd2.in‚0' disk.nvme0c0n1.out‚57654.2783203125" disk.dm-0.out‚57654.2783203125 disk.dm-1.out‚235.875 cpu‚25.32814094130798 disk.rbd1.out‚ 3.06640625 disk.sda2.in‚0 disk.sda3.out‚235.875# disk.nvme0c0n1.in‚ 1754.82421875V‰92‡- _wandbruntime‚6391  _runtime‚6391œ~ÔnɈà ' lr-AdamW/pg2‚0.00014400090294987893 trainer/global_step‚7549 % _timestamp‚1.7884439515277996e+09 _runtime‚6391.572639368 _step‚618 ' lr-AdamW/pg1‚0.00014400090294987893[ÛõÎ‰È ' lr-AdamW/pg1‚0.00014400090294987893 ' lr-AdamW/pg2‚0.00014400090294987893 trainer/global_step‚7549 % _timestamp‚1.7884439515277996e+09 _runtime‚6391.572639368 _step‚618êÔuÅG2Š- _wandbruntime‚6391  _runtime‚63919žà‹ŠP RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 _runtime‚6391.573530407 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52412379 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14016677519360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _step‚619 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111607289 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14016677519360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842861080951808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14014077309952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODÌ2ƒ‡0eviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140905825 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚856877758471168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚856877758471168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72429683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721014174131200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842861054560256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52411937 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13990834520688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚856875131870208 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721014145996800 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140905825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111607283 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842861080951808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843363399019500 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13993432805356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124841620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14014077309952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740211910283776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740211837850112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19197691853312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843360775306864 % _timestamp‚1.7884439515287118e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111607289 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111607283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72429686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074ˆÔR©(ŒP =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140905825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111607283 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842861080951808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843363399019500 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13993432805356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124841620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14014077309952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740211910283776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740211837850112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19197691853312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843360775306864 % _timestamp‚1.7884439515287118e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111607289 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111607283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72429686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 _runtime‚6391.573530407 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52412379 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14016677519360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _step‚619 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111607289 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14016677519360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842861080951808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14014077309952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140905825 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚856877758471168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚856877758471168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72429683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721014174131200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842861054560256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52411937 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13990834520688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚856875131870208 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721014145996800 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408ë_¨k2- _wandbruntime‚6392  _runtime‚6392$y®'Ž”N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52418516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140944119 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140924059 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19200223341568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72438879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124857822 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111621519 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829476281960448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72438882 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚6392.314892909 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842968994265600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19200269724672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829476307388416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14015909638656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111621525 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124857395 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚856987530518528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14018509861376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842969020657152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252545578 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721106562184704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14015909638656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111621525 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13992663863356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140924059 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843468945823804 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545578 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884439522696671e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13992663863356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140924059 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843468945823804 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252545578 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884439522696671e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124857822 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111621519 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829476281960448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72438882 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚6392.314892909 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842968994265600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19200269724672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829476307388416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14015909638656 ZMDevÕ[AžÓ iceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111621525 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124857395 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚856987530518528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14018509861376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649ì €ü2- _wandbruntime‚6392  _runtime‚6392.’…%]‘× 2loss/FBt4/assignment_loss‚0.007147134708551069 1loss/SRqqt4/detection_loss‚1.9239599158987404 0loss/FRt1/assignment_loss‚1.5328942939292554 0loss/FRt2/assignment_loss‚3.1228480268092382 1loss/SRqqt1/assignment_loss‚1.168759711022521 1loss/FBt1/assignment_loss‚0.16093293063762956 .loss/FRt4/detection_loss‚2.073024096641135 1loss/SRqqt2/detection_loss‚1.8353986583002235 _step‚621 0loss/FBt2/assignment_loss‚0.2998729760542379 &loss/total_loss‚42.449913122889726 epoch‚6 /loss/FBt1/detection_loss‚0.6544622982921148 .loss/FBt2/detection_loss‚3.529230909688132 0loss/SRqqt1/detection_loss‚1.152895665181608 _runtime‚6392.315991149 /loss/FRt3/detection_loss‚2.5881306063930616 /loss/FBt3/detection_loss‚3.3739060908555984 /loss/FBt4/detection_loss‚2.5174324380026922 trainer/global_step‚7549 /loss/FRt3/assignment_loss‚3.560816977963303 2loss/SRqqt4/assignment_loss‚1.1730977249075294 3loss/FBt3/assignment_loss‚0.0020817467202505213 .loss/FRt2/detection_loss‚2.477554816575277 2loss/SRqqt2/assignment_loss‚1.1779829851635124 /loss/FRt1/detection_loss‚1.8643643443667612 0loss/SRqqt3/detection_loss‚2.361563517898321 # _timestamp‚1.78844395227142e+09 0loss/FRt4/assignment_loss‚2.4914484315730157 2loss/SRqqt3/assignment_loss‚1.4001068253070117”å1„b’Ü /loss/FRt3/detection_loss‚2.5881306063930616 /loss/FBt3/detection_loss‚3.3739060908555984 /loss/FBt4/detection_loss‚2.5174324380026922 trainer/global_step‚7549 /loss/FRt3/assignment_loss‚3.560816977963303 2loss/SRqqt4/assignment_loss‚1.1730977249075294 3loss/FBt3/assignment_loss‚0.0020817467202505213 .loss/FRt2/detection_loss‚2.477554816575277 2loss/SRqqt2/assignment_loss‚1.1779829851635124 /loss/FRt1/detection_loss‚1.8643643443667612 0loss/SRqqt3/detection_loss‚2.361563517898321 # _timestamp‚1.78844395227142e+09 0loss/FRt4/assignment_loss‚2.4914484315730157 2loss/SRqqt3/assignment_loss‚1.4001068253070117 2loss/FBt4/assignment_loss‚0.007147134708551069 1loss/SRqqt4/detection_loss‚1.9239599158987404 0loss/FRt1/assignment_loss‚1.5328942939292554 0loss/FRt2/assignment_loss‚3.1228480268092382 1loss/SRqqt1/assignment_loss‚1.168759711022521 1loss/FBt1/assignment_loss‚0.16093293063762956 .loss/FRt4/detection_loss‚2.073024096641135 1loss/SRqqt2/detection_loss‚1.8353986583002235 _step‚621 0loss/FBt2/assignment_loss‚0.2998729760542379 &loss/total_loss‚42.449913122889726 epoch‚6 /loss/FBt1/detection_loss‚0.6544622982921148 .loss/FBt2/detection_loss‚3.529230909688132 0loss/SRqqt1/detection_loss‚1.152895665181608 _runtime‚6392.315991149íL;tnŸ“‚(:” ¼úåÔ¹ñ„É gpu.0.gpu‚59.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚261.917' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.383399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë[gïB”‚(:· ¼úåÔÕš†Ë disk.dm-0.in‚ 1754.82421875 disk.rbd2.out‚539.9638671875 disk.sda.out‚ 236.16796875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1754.82421875$ disk./.usageGB‚1108.193115234375 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚57654.87841796875! proc.memory.rssMB‚ 2629.046875 cpu‚25.327497880913864 network.sent‚ 1.8808705e+07$ memory_percent‚89.14093971252441 proc.cpu.threads‚17 network.recv‚ 2.326395e+06) disk./.usagePercent‚74.37176979933598 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚57654.87841796875 disk.rbd0.out‚0( proc.memory.availableMB‚ 1779.1484375 disk.rbd1.out‚ 3.06640625 disk.sda1.in‚0# disk.dm-0.out‚57654.87841796875* proc.memory.percent‚16.046428680419922 disk.sda3.out‚ 236.16796875 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 236.16796875 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875Í)­Ÿ•‚(:” ËúåÔ”ÌÙÈ gpu.0.gpu‚94.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.807' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.561400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕM9–‚(:® ËúåÔ­ÚË disk.rbd0.in‚0 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.sda3.out‚ 236.71875& disk.nvme0n1.out‚57752.32373046875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚57752.32373046875 network.sent‚ 1.8918377e+07% disk./.usageGB‚1108.1931190490723 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.06640625 disk.sda2.in‚0 disk.rbd2.out‚539.9912109375 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875 network.recv‚ 2.333218e+06! disk.nvme0n1.in‚ 1754.82421875) proc.memory.availableMB‚ 1779.15234375 disk.rbd3.out‚0# memory_percent‚89.1409158706665* proc.memory.percent‚16.046428680419922 disk.sda.out‚ 236.71875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 236.71875 cpu‚25.326846964349723 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1754.82421875# disk.dm-0.out‚57752.32373046875! proc.memory.rssMB‚ 2629.046875 disk.sda1.out‚0( disk./.usagePercent‚74.3717700553435 disk.rbd1.in‚0¿žè2—- _wandbruntime‚6430  _runtime‚6430>~üöɘà _step‚622 ' lr-AdamW/pg1‚0.00014130287719012286 ' lr-AdamW/pg2‚0.00014130287719012286 trainer/global_step‚7599 % _timestamp‚1.7884439905313683e+09 _runtime‚6430.576065198Î,wçÎ™È ' lr-AdamW/pg1‚0.00014130287719012286 ' lr-AdamW/pg2‚0.00014130287719012286 trainer/global_step‚7599 % _timestamp‚1.7884439905313683e+09 _runtime‚6430.576065198 _step‚622î=jU–2š- _wandbruntime‚6430  _runtime‚6430E’5(›‰P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862363747986432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848258066219008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744957756538880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚848771938457708 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14084913529964 _runtime‚ 6430.57694741 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚834687024927744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112319089 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚7599 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112319083 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚848258039827456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112319089 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚725633576806400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚834686999499776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862363747986432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14082315245296 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744957682008064 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141820877 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 _step‚623 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862366374587392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254160000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚848769314745072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 % _timestamp‚1.7884439905321062e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141840911 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141820877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72889486 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725633548672000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72889483 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14105708158976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19324133336064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52745215 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19324179732480 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125634256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÜUL{(œŽP XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14105708158976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19324133336064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52745215 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19324179732480 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125634256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862363747986432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848258066219008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744957756538880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚848771938457708 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14084913529964 _runtime‚ 6430.57694741 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚834687024927744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112319089 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚7599 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112319083 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚848258039827456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112319089 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚725633576806400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚834686999499776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862363747986432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14082315245296 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744957682008064 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141820877 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 _step‚623 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862366374587392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254160000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚848769314745072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 % _timestamp‚1.7884439905321062e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141840911 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141820877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72889486 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725633548672000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72889483 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034ï _ž2- _wandbruntime‚6431  _runtime‚6431Ńosà ž”N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725725936725504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14110140710400 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725725964859904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6431.318982107 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19326697722368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14110140710400 ZFDevic<š²úWeStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚848365979532800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112333325 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14107540487680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125650092 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72898679 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7599 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112333319 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚745052662582272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚848880108974856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141859171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 _step‚624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚862476146634752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚848365979532800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚848366005924352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19326651339264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848366005924352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚862473520020480 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141859171 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14107540487680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72898682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚834793340674048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112333319 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚862476146634752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52751413 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14084144587964 % _timestamp‚1.7884439912736652e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112333325 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚745052588064768 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141839111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚848877485262012 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚834793366102016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376È´'Ÿ™N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚834793366102016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725725936725504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14110140710400 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725725964859904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6431.318982107 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19326697722368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14110140710400 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚848365979532800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112333325 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14107540487680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125650092 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72898679 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7599 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112333319 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚745052662582272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚848880108974856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141859171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 _step‚624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚862476146634752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚848365979532800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚848366005924352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19326651339264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848366005924352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚862473520020480 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141859171 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14107540487680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72898682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚834793340674048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112333319 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚862476146634752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52751413 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14084144587964 % _timestamp‚1.7884439912736652e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112333325 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚745052588064768 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141839111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚848877485262012 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ðR6¢’2 - _wandbruntime‚6431  _runtime‚6431 »PJZ¡Ô epoch‚6 0loss/FRt4/assignment_loss‚2.6572563290596007 2loss/SRqqt2/assignment_loss‚1.3554327176154957 1loss/SRqqt4/assignment_loss‚1.412789907817747 /loss/FBt1/assignment_loss‚0.084499210869596 1loss/FBt4/assignment_loss‚0.09846572095993907 1loss/SRqqt2/detection_loss‚1.7919339029037433 0loss/SRqqt4/detection_loss‚1.680759084867496 trainer/global_step‚7599 /loss/FBt2/detection_loss‚2.3426184143338884 %loss/total_loss‚41.19723529672541 _runtime‚6431.320420468 0loss/FRt1/assignment_loss‚1.2351441383361816 .loss/FRt2/detection_loss‚2.407615455920282 _step‚625 0loss/FRt2/assignment_loss‚2.4184897023698557 /loss/FRt1/detection_loss‚2.0360095139879446 1loss/SRqqt1/detection_loss‚1.1296724592198113 2loss/SRqqt3/assignment_loss‚1.8694239378507649 .loss/FRt3/detection_loss‚2.643278450622837 % _timestamp‚1.7884439912758446e+09 2loss/SRqqt1/assignment_loss‚1.1052934478235539 1loss/FBt2/assignment_loss‚0.01923528639599681 1loss/SRqqt3/detection_loss‚2.5544456328664507 /loss/FBt1/detection_loss‚0.6170373198565566 .loss/FBt3/detection_loss‚2.613845637866429 0loss/FRt3/assignment_loss‚3.8052101822999806 1loss/FBt3/assignment_loss‚0.14068379885117924 -loss/FRt4/detection_loss‚2.51394850148095 /loss/FBt4/detection_loss‚2.6641465425491333ÑÉ_¢Ù /loss/FBt1/assignment_loss‚0.084499210869596 1loss/FBt4/assignment_loss‚0.09846572095993907 1loss/SRqqt2/detection_loss‚1.7919339029037433 0loss/SRqqt4/detection_loss‚1.680759084867496 trainer/global_step‚7599 /loss/FBt2/detection_loss‚2.3426184143338884 %loss/total_loss‚41.19723529672541 _runtime‚6431.320420468 0loss/FRt1/assignment_loss‚1.2351441383361816 .loss/FRt2/detection_loss‚2.407615455920282 _step‚625 0loss/FRt2/assignment_loss‚2.4184897023698557 /loss/FRt1/detection_loss‚2.0360095139879446 1loss/SRqqt1/detection_loss‚1.1296724592198113 2loss/SRqqt3/assignment_loss‚1.8694239378507649 .loss/FRt3/detection_loss‚2.643278450622837 % _timestamp‚1.7884439912758446e+09 2loss/SRqqt1/assignment_loss‚1.1052934478235539 1loss/FBt2/assignment_loss‚0.01923528639599681 1loss/SRqqt3/detection_loss‚2.5544456328664507 /loss/FBt1/detection_loss‚0.6170373198565566 .loss/FBt3/detection_loss‚2.613845637866429 0loss/FRt3/assignment_loss‚3.8052101822999806 1loss/FBt3/assignment_loss‚0.14068379885117924 -loss/FRt4/detection_loss‚2.51394850148095 /loss/FBt4/detection_loss‚2.6641465425491333 epoch‚6 0loss/FRt4/assignment_loss‚2.6572563290596007 2loss/SRqqt2/assignment_loss‚1.3554327176154957 1loss/SRqqt4/assignment_loss‚1.412789907817747ñ±ö™|Ÿ£‚(:” ÚúåÔçÖºÉ gpu.0.gpu‚72.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚239.472' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.894400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G&`?¤‚(:´ ÚúåÔÚ²ÇÍ( disk./.usagePercent‚74.3717662152305 disk.dm-1.out‚ 237.04296875 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.82421875" disk.dm-0.out‚57752.7958984375 disk.sda1.in‚0" proc.memory.rssMB‚ 2629.1328125 cpu‚25.32618911221775# disk.nvme0c0n1.in‚ 1754.82421875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 network.recv‚ 2.337058e+06% disk.nvme0n1.out‚57752.7958984375' disk.nvme0c0n1.out‚57752.7958984375 disk.rbd1.out‚ 3.06640625 disk.sda.in‚0.5625 disk.sda.out‚ 237.04296875 disk.sda1.out‚0 disk.sda2.in‚0% disk./.usageGB‚1108.1930618286133 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0$ memory_percent‚89.14198875427246( proc.memory.availableMB‚ 1778.9765625* proc.memory.percent‚16.046953201293945 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.out‚ 237.04296875 disk.rbd3.in‚0 network.sent‚ 1.8923707e+07 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚541.5810546875\ÊÉž¥‚(:“ éúåÔ¦öàÈ gpu.0.gpu‚93.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fýÝ´>¦‚(:³ éúåÔˈµË disk.dm-1.out‚ 237.64453125 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 237.64453125% disk./.usageGB‚1108.1931190490723 disk.rbd2.out‚541.6083984375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.sda.in‚0.5625 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.82421875% disk.nvme0n1.out‚57850.2802734375" disk.dm-0.out‚57850.2802734375# memory_percent‚89.1413688659668' proc.memory.availableMB‚ 1779.078125 disk.sda1.in‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875 cpu‚25.325528389451556 disk.rbd3.out‚0# proc.memory.rssMB‚ 2629.22265625 network.sent‚ 1.9032977e+07 disk.rbd2.in‚0' disk.nvme0c0n1.out‚57850.2802734375) proc.memory.percent‚16.04750156402588( disk./.usagePercent‚74.3717700553435 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 237.64453125 disk.rbd0.out‚0 network.recv‚ 2.343467e+06 disk.rbd1.out‚ 3.06640625 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.82421875Ùç4ÙŸ§‚(:” øúåÔ‚þòÈ gpu.0.gpu‚62.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚172.019' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.403800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#´„;¨‚(:° øúåÔé±£Ë disk.sda1.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚57851.15478515625( disk.nvme0c0n1.out‚57851.15478515625 disk.dm-1.out‚ 238.015625 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.dm-0.in‚ 1754.82421875# proc.memory.rssMB‚ 2629.22265625 network.sent‚ 1.903881e+07 disk.drbd1777.out‚0$ memory_percent‚89.14132118225098# disk.dm-0.out‚57851.15478515625 network.recv‚ 2.347637e+06 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1754.82421875 cpu‚25.32485429892205 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.dm-1.in‚0.5625( proc.memory.availableMB‚ 1779.0859375) proc.memory.percent‚16.04750156402588 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚543.2685546875 disk.sda3.in‚0.5625 disk.rbd0.in‚0% disk./.usageGB‚1108.1928176879883 disk.sda.out‚ 238.015625 disk.rbd3.in‚0 disk.sda3.out‚ 238.015625 disk.rbd0.out‚0( disk./.usagePercent‚74.3717498307483 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.06640625¿YIß2©- _wandbruntime‚6469  _runtime‚6469DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚854180477895916 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867852364102656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142735929 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73349283 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113030889 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126429148 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113030883 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14197339008000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 6469.73381141 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14173795969904 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255786852 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚854177854183280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53080328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749703253117952 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14199939217408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚840004083641344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729ïìóal 555456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53079865 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730252979481600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867854990703616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚853655051486208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255786852 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚854177854183280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53080328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749703253117952 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14199939217408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚840004083641344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53079865 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730252979481600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867854990703616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚853655051486208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚854180477895916 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867852364102656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142735929 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400óØPÚM2¯- _wandbruntime‚6470  _runtime‚6470ó¤„'°“N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730345367535104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚840110424815616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14178223597448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142774223 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53086474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142774223 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255799282 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113045119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚840110399387648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730345339400704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53086918 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚854286024700220 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19452818950144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867964762750976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73358482 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113045125 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚853762964800000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6470.474526427 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚853762991191552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14201771559424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚749798232868352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14175625312572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113045119 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14199171336704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126444953 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _step‚628 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142754163 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14201771559424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575+¡†Ý%±˜N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚853762964800000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126445400 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19452818950144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867964762750976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73358482 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113045125 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚853762964800000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6470.474526427 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚853762991191552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14201771559424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚749798232868352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14175625312572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113045119 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14199171336704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126444953 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _step‚628 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142754163 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14201771559424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730345367535104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚840110424815616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14178223597448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142774223 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53086474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142774223 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255799282 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113045119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚840110399387648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730345339400704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53086918 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚854286024700220 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F\³Ö /loss/FRt4/detection_loss‚2.4046472364001805 .loss/FBt2/detection_loss‚2.384004533290863 0loss/FRt1/assignment_loss‚0.8907490332018245 2loss/SRqqt1/assignment_loss‚0.8352205893024802 1loss/SRqqt2/detection_loss‚1.8675604524477474 %loss/total_loss‚41.49971786428536 2loss/SRqqt3/assignment_loss‚1.5921364986608106 .loss/FRt3/detection_loss‚2.261253478765292 .loss/FBt1/detection_loss‚0.624637066812059 /loss/FBt3/detection_loss‚4.8356258273124695 .loss/FBt4/detection_loss‚2.367836507883939 /loss/FRt2/assignment_loss‚2.711297993252917 2loss/SRqqt2/assignment_loss‚1.4910018930896975 1loss/FBt1/assignment_loss‚0.12651129299615507 $ _timestamp‚1.788444030431236e+09 1loss/FBt3/assignment_loss‚0.12479428711230867 /loss/FRt1/detection_loss‚1.4668230671774258 1loss/SRqqt4/detection_loss‚1.7751545149087906 .loss/FRt2/detection_loss‚2.372768674681826 2loss/SRqqt4/assignment_loss‚0.9545324483129661 1loss/FBt2/assignment_loss‚0.22918139045359567 1loss/SRqqt3/detection_loss‚2.5381207389216267 _runtime‚6470.475842848 0loss/FRt3/assignment_loss‚3.6747622521300065 0loss/FRt4/assignment_loss‚2.6693198561668394 trainer/global_step‚7649 1loss/FBt4/assignment_loss‚0.39425171264420816 1loss/SRqqt1/detection_loss‚0.9075265183593287 epoch‚6 _step‚629‰:æya´Û 1loss/FBt2/assignment_loss‚0.22918139045359567 1loss/SRqqt3/detection_loss‚2.5381207389216267 _runtime‚6470.475842848 0loss/FRt3/assignment_loss‚3.6747622521300065 0loss/FRt4/assignment_loss‚2.6693198561668394 trainer/global_step‚7649 1loss/FBt4/assignment_loss‚0.39425171264420816 1loss/SRqqt1/detection_loss‚0.9075265183593287 epoch‚6 _step‚629 /loss/FRt4/detection_loss‚2.4046472364001805 .loss/FBt2/detection_loss‚2.384004533290863 0loss/FRt1/assignment_loss‚0.8907490332018245 2loss/SRqqt1/assignment_loss‚0.8352205893024802 1loss/SRqqt2/detection_loss‚1.8675604524477474 %loss/total_loss‚41.49971786428536 2loss/SRqqt3/assignment_loss‚1.5921364986608106 .loss/FRt3/detection_loss‚2.261253478765292 .loss/FBt1/detection_loss‚0.624637066812059 /loss/FBt3/detection_loss‚4.8356258273124695 .loss/FBt4/detection_loss‚2.367836507883939 /loss/FRt2/assignment_loss‚2.711297993252917 2loss/SRqqt2/assignment_loss‚1.4910018930896975 1loss/FBt1/assignment_loss‚0.12651129299615507 $ _timestamp‚1.788444030431236e+09 1loss/FBt3/assignment_loss‚0.12479428711230867 /loss/FRt1/detection_loss‚1.4668230671774258 1loss/SRqqt4/detection_loss‚1.7751545149087906 .loss/FRt2/detection_loss‚2.372768674681826 2loss/SRqqt4/assignment_loss‚0.9545324483129661õ úLÇ“µ‚(:ˆ ‡ûåÔáù£É gpu.0.gpu‚66.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.713 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Nµ©>¶‚(:³ ‡ûåÔéÿÌ disk.dm-0.in‚ 1754.82421875 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.dm-1.out‚ 238.66796875 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.sent‚ 1.9045096e+07) disk./.usagePercent‚74.37178899990103 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1754.82421875# proc.memory.rssMB‚ 2629.22265625 cpu‚25.324176351834808 disk.sda2.in‚0% disk.nvme0n1.out‚57851.5048828125 disk.rbd1.out‚ 3.06640625$ memory_percent‚89.14237022399902) proc.memory.percent‚16.04750156402588 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚544.8037109375 network.recv‚ 2.35205e+06 disk.sda3.in‚0.5625 disk.sda.out‚ 238.66796875 disk.sda1.in‚0' disk.nvme0c0n1.out‚57851.5048828125( proc.memory.availableMB‚ 1778.9140625# disk./.usageGB‚1108.19340133667" disk.dm-0.out‚57851.5048828125 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 238.66796875 disk.rbd0.in‚0¢pûÉ”·‚(:‰ –ûåÔãø€Ë gpu.0.gpu‚84.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ŒÈ@@¸‚(:µ –ûåÔ¼ê‚Í disk.sda2.out‚0 cpu‚25.32353760486064 proc.cpu.threads‚17 network.recv‚ 2.358714e+06 disk.rbd0.out‚0 disk.sda2.in‚0 network.sent‚ 1.9154079e+07 disk.rbd3.in‚0% disk.nvme0n1.out‚57949.5087890625 disk.sda3.out‚ 238.98046875 disk.dm-1.out‚ 238.98046875$ memory_percent‚89.14179801940918' disk.nvme0c0n1.out‚57949.5087890625 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda.out‚ 238.98046875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.82421875 disk.drbd1777.out‚0% disk./.usageGB‚1108.1933670043945 disk.drbd1777.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1779.0078125# proc.memory.rssMB‚ 2629.22265625! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875" disk.dm-0.out‚57949.5087890625 disk.rbd1.in‚0 disk.rbd2.out‚544.9443359375) proc.memory.percent‚16.04750156402588) disk./.usagePercent‚74.37178669583324 disk.rbd0.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda.in‚0.5625¥"ã2¹- _wandbruntime‚6508  _runtime‚650812ìÊɺà ' lr-AdamW/pg1‚0.00013594017976817504 ' lr-AdamW/pg2‚0.00013594017976817504 trainer/global_step‚7699 % _timestamp‚1.7884440685861838e+09 _runtime‚6508.630863954 _step‚630æd«èÎ»È _step‚630 ' lr-AdamW/pg1‚0.00013594017976817504 ' lr-AdamW/pg2‚0.00013594017976817504 trainer/global_step‚7699 % _timestamp‚1.7884440685861838e+09 _runtime‚6508.630863954ö9»°<2¼- _wandbruntime‚6508  _runtime‚6508ï ™(½ŠP WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113742683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚734872382156800 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113742689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143650981 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14288969857024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚734872354022400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19576063513088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845321142354944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859052010361856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127223943 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14265276694512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14267874979180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚873340980218880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884440685869496e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143671015 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19576015019520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 trainer/global_step‚7699 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859052010361856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 _step‚631 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚873343606819840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113742689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14291570066432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143650981 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53414857 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚873343606819840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14288969857024 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127223480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53414397 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845321116926976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _runtime‚6508.631822991 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113742683 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859052036753408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143671015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73809083 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚754448369041920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚859589017334124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚859586393621488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚7544484456698883u€(¾P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53414857 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚873343606819840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14288969857024 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127223480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53414397 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845321116926976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _runtime‚6508.631822991 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113742683 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859052036753408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143671015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73809083 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚754448369041920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚859589017334124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚859586393621488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚754448445669888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113742683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚734872382156800 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113742689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143650981 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14288969857024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚734872354022400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19576063513088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845321142354944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859052010361856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127223943 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14265276694512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14267874979180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚873340980218880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884440685869496e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143671015 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19576015019520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 trainer/global_step‚7699 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859052010361856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 _step‚631 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚873343606819840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113742689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257413704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14291570066432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143650981 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649÷ÇId2¿- _wandbruntime‚6509  _runtime‚6509ñp6ŸÀ”N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14290802185728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19578578945536 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73818282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113756919 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127239192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚734964742075904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113756925 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14293402408448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859159950067200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚873450752252928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127239622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859694564138428 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 % _timestamp‚1.7884440693201487e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚873453378867200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143669215 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14290802185728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚845427483529216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14267106037180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _runtime‚6509.365375344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859159976458752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19578532562432 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14293402408448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143669215 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14269704322056 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143689275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚754543349155840 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113756919 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143669215 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14290802185728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚845427483529216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14267106037180 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _runtime‚6509.365375344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859159976458752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19578532562432 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14293402408448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143669215 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14269704322056 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143689275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚754543349155840 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113756919 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127239192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚734964742075904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113756925 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14293402408448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859159950067200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚873450752252928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127239622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859694564138428 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 % _timestamp‚1.7884440693201487e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776øî*Øg2Â- _wandbruntime‚6509  _runtime‚6509;pªYÃÓ /loss/FRt2/assignment_loss‚2.913260314025377 /loss/FRt1/detection_loss‚1.6561779748628158 1loss/SRqqt2/detection_loss‚1.9015248402332265 .loss/FBt1/detection_loss‚0.547852958111244 2loss/SRqqt2/assignment_loss‚1.7231143036236365 1loss/FBt2/assignment_loss‚0.07799507061135955 /loss/FBt3/detection_loss‚2.3596178208078658 %loss/total_loss‚39.05102319413021 epoch‚6 trainer/global_step‚7699 _runtime‚ 6509.36642409 _step‚633 0loss/FRt4/assignment_loss‚2.4517088612994633 2loss/SRqqt4/assignment_loss‚1.0848146247987946 /loss/FRt3/assignment_loss‚2.940765960796459 1loss/SRqqt1/assignment_loss‚1.090471539582598 1loss/FBt3/assignment_loss‚0.18358422112318554 1loss/SRqqt1/detection_loss‚0.9099609617092838 1loss/SRqqt4/detection_loss‚1.6785861923048893 /loss/FBt4/detection_loss‚2.7526436150074005 0loss/FRt1/assignment_loss‚0.9019772552925608 /loss/FBt2/detection_loss‚2.5024966955184937 1loss/FBt1/assignment_loss‚0.15496213960027866 1loss/FBt4/assignment_loss‚0.25609552111859557 /loss/FRt2/detection_loss‚2.5960943110679326 /loss/FRt4/detection_loss‚2.6282102131360285 # _timestamp‚1.78844406932188e+09 1loss/SRqqt3/assignment_loss‚1.163869660543768 .loss/FRt3/detection_loss‚2.386257818388372 0loss/SRqqt3/detection_loss‚2.188980320566579:<í^ÄØ /loss/FRt3/assignment_loss‚2.940765960796459 1loss/SRqqt1/assignment_loss‚1.090471539582598 1loss/FBt3/assignment_loss‚0.18358422112318554 1loss/SRqqt1/detection_loss‚0.9099609617092838 1loss/SRqqt4/detection_loss‚1.6785861923048893 /loss/FBt4/detection_loss‚2.7526436150074005 0loss/FRt1/assignment_loss‚0.9019772552925608 /loss/FBt2/detection_loss‚2.5024966955184937 1loss/FBt1/assignment_loss‚0.15496213960027866 1loss/FBt4/assignment_loss‚0.25609552111859557 /loss/FRt2/detection_loss‚2.5960943110679326 /loss/FRt4/detection_loss‚2.6282102131360285 # _timestamp‚1.78844406932188e+09 1loss/SRqqt3/assignment_loss‚1.163869660543768 .loss/FRt3/detection_loss‚2.386257818388372 0loss/SRqqt3/detection_loss‚2.188980320566579 /loss/FRt2/assignment_loss‚2.913260314025377 /loss/FRt1/detection_loss‚1.6561779748628158 1loss/SRqqt2/detection_loss‚1.9015248402332265 .loss/FBt1/detection_loss‚0.547852958111244 2loss/SRqqt2/assignment_loss‚1.7231143036236365 1loss/FBt2/assignment_loss‚0.07799507061135955 /loss/FBt3/detection_loss‚2.3596178208078658 %loss/total_loss‚39.05102319413021 epoch‚6 trainer/global_step‚7699 _runtime‚ 6509.36642409 _step‚633 0loss/FRt4/assignment_loss‚2.4517088612994633 2loss/SRqqt4/assignment_loss‚1.0848146247987946ù2Ûú”Å‚(:‰ ¥ûåÔÎç€É gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚232.209' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó$R„?Æ‚(:´ ¥ûåÔãàÄË disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.15234375& proc.memory.availableMB‚ 1779.03125# proc.memory.rssMB‚ 2629.22265625) disk./.usagePercent‚74.37179079195377 disk.sda2.out‚0! disk.nvme0n1.in‚ 1754.82421875" disk.dm-0.out‚57949.8173828125 disk.rbd2.in‚0$ memory_percent‚89.14165496826172 cpu‚25.322921833251442 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.362968e+06 disk.sda2.in‚0 disk.dm-0.in‚ 1754.82421875 disk.sda.in‚0.5625% disk.nvme0n1.out‚57949.8173828125' disk.nvme0c0n1.out‚57949.8173828125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 network.sent‚ 1.9160009e+07 disk.rbd2.out‚546.4794921875) proc.memory.percent‚16.04750156402588% disk./.usageGB‚1108.1934280395508 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.dm-1.out‚ 239.58984375 disk.drbd1777.in‚0 disk.sda.out‚ 239.58984375 disk.sda3.out‚ 239.58984375[9½ŸÇ‚(:” ´ûåÔè’½É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚262.207' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.441399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¤hø6È‚(:« ´ûåÔ•ì«Ë disk.rbd2.in‚0 disk.rbd2.out‚546.5576171875( proc.memory.availableMB‚ 1778.9296875 cpu‚25.322298638631366! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.in‚0.5625 disk.dm-1.out‚239.875 disk.rbd1.out‚ 3.21484375$ memory_percent‚89.14227485656738# proc.memory.rssMB‚ 2630.39453125* proc.memory.percent‚16.054654121398926) disk./.usagePercent‚74.37178771986336 disk.sda.in‚0.5625 disk.sda.out‚239.875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1108.1933822631836 disk.dm-0.in‚ 1754.82421875& disk.nvme0n1.out‚58047.37060546875 network.recv‚ 2.369939e+06 disk.sda2.out‚0( disk.nvme0c0n1.out‚58047.37060546875 network.sent‚ 1.9269292e+07 disk.sda3.out‚239.875# disk.nvme0c0n1.in‚ 1754.82421875 proc.cpu.threads‚17# disk.dm-0.out‚58047.37060546875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.in‚0× ýòŸÉ‚(:” ÃûåÔ¾ÿ£É gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚104.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.962799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÂOAÊ‚(:¶ ÃûåÔôé›Ë( disk.nvme0c0n1.out‚58047.67919921875 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.out‚0) proc.memory.availableMB‚ 1779.20703125" proc.memory.rssMB‚ 2631.4609375 network.sent‚ 1.9275667e+07 disk.sda.out‚ 240.41015625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 240.41015625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.82421875# disk.dm-0.out‚58047.67919921875 proc.cpu.threads‚17% disk./.usageGB‚1108.1934547424316 disk.sda2.out‚0& disk.nvme0n1.out‚58047.67919921875 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 cpu‚25.3216776430293 disk.dm-0.in‚ 1754.82421875 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.21484375 disk.sda2.in‚0) disk./.usagePercent‚74.37179258400651 disk.rbd2.out‚548.0966796875$ memory_percent‚89.14058208465576( proc.memory.percent‚16.0611629486084 network.recv‚ 2.374666e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 240.41015625<ϲÁ2Ë- _wandbruntime‚6547  _runtime‚6547•?¿æÇÌÁ & lr-AdamW/pg2‚0.0001332766256840882 trainer/global_step‚7749 % _timestamp‚1.7884441076888087e+09 _runtime‚6547.733498225 _step‚634 & lr-AdamW/pg1‚0.0001332766256840882FsèïÌÍÆ & lr-AdamW/pg1‚0.0001332766256840882 & lr-AdamW/pg2‚0.0001332766256840882 trainer/global_step‚7749 % _timestamp‚1.7884441076888087e+09 _runtime‚6547.733498225 _step‚634ú V2Î- _wandbruntime‚6547  _runtime‚6547 ϪžŽ'ωP D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53750793 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚864997556772332 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14380600706048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14380600706048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚878832222936064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚864448995629056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚878829596335104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 _step‚635 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114454483 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚739491756697600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19701872826368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚864449022020608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739491784832000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 259040556 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6547.734483933 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14383200915456 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚864449022020608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144566033 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128020145 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128019676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114454489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144586067 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53751259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚850638175640576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚864448995629056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74268883 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14356757419120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 trainer/global_step‚7749 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚759193657658368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚759193581030400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114454483 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚878829596335104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144566033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚878832222936064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144586067 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14359355703788 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚864994933059696 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚850638201068544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74268886 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/aÞgºllocated_bytes.small_pool.allocated‚14383200915456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575í乧(ÐŽP H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144566033 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128020145 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128019676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114454489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144586067 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53751259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚850638175640576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚864448995629056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74268883 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14356757419120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 trainer/global_step‚7749 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚759193657658368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚759193581030400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114454483 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚878829596335104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144566033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚878832222936064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144586067 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14359355703788 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚864994933059696 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚850638201068544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74268886 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14383200915456 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53750793 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚864997556772332 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14380600706048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14380600706048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚878832222936064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚864448995629056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚878829596335104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 _step‚635 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114454483 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚469 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚739491756697600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19701872826368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚864449022020608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739491784832000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 259040556 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6547.734483933 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14383200915456 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚864449022020608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248ûˆK Ñ2Ñ- _wandbruntime‚6548  _runtime‚6548äLP'Ò”N EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114468719 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144604327 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚865103103576636 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚878939368369152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74278079 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114468725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144604327 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128036028 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚864556961725952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128035573 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19704336815104 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _runtime‚6548.470855155 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739584172885504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚739584144751104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14382433034752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚850744542242816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114468719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53757494 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74278082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚864556935334400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144584267 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚759288481566208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14385033257472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚850744516814848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400r»à£'Ó™N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚864556961725952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128035573 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19704336815104 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _runtime‚6548.470855155 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739584172885504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚739584144751104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14382433034752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚850744542242816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114468719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53757494 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74278082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚864556935334400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144584267 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚759288481566208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14385033257472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚850744516814848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114468719 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144604327 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚865103103576636 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚878939368369152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74278079 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114468725 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144604327 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128036028 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649üÒ 2Ô- _wandbruntime‚6548  _runtime‚6548gˆÙ€aÕÛ /loss/FBt2/detection_loss‚2.4541541039943695 /loss/FRt1/detection_loss‚1.5115625495836138 2loss/SRqqt3/assignment_loss‚1.5649402955126377 _step‚637 /loss/FRt3/assignment_loss‚3.394915042604719 1loss/SRqqt2/assignment_loss‚1.586247556758198 0loss/SRqqt2/detection_loss‚2.166608710519292 /loss/FBt4/detection_loss‚3.2731491128603616 0loss/FRt2/assignment_loss‚2.6347777631547715 2loss/SRqqt4/assignment_loss‚1.5186043815179304 1loss/SRqqt1/detection_loss‚0.9530133666736739 1loss/SRqqt4/detection_loss‚1.5717364918101917 /loss/FBt1/detection_loss‚0.6644003912444679 /loss/FBt3/detection_loss‚3.0227956026792526 _runtime‚6548.471907598 1loss/FBt1/assignment_loss‚0.06467878418399117 epoch‚6 trainer/global_step‚7749 %loss/total_loss‚41.37571375043427 $ _timestamp‚1.788444108427325e+09 1loss/FBt4/assignment_loss‚0.06186092823433379 /loss/FRt2/detection_loss‚2.4043074978722467 0loss/FRt1/assignment_loss‚1.2971999371051788 0loss/FRt4/assignment_loss‚2.8223484739661218 2loss/SRqqt1/assignment_loss‚0.8716233101706686 3loss/FBt3/assignment_loss‚0.0016921018541324884 1loss/FBt2/assignment_loss‚0.05639138499585291 /loss/FRt3/detection_loss‚2.3915921965497544 /loss/FRt4/detection_loss‚2.4983855463564395 1loss/SRqqt3/detection_loss‚2.5887282202320714Xˆb‚5Öà 0loss/SRqqt2/detection_loss‚2.166608710519292 /loss/FBt4/detection_loss‚3.2731491128603616 0loss/FRt2/assignment_loss‚2.6347777631547715 2loss/SRqqt4/assignment_loss‚1.5186043815179304 1loss/SRqqt1/detection_loss‚0.9530133666736739 1loss/SRqqt4/detection_loss‚1.5717364918101917 /loss/FBt1/detection_loss‚0.6644003912444679 /loss/FBt3/detection_loss‚3.0227956026792526 _runtime‚6548.471907598 1loss/FBt1/assignment_loss‚0.06467878418399117 epoch‚6 trainer/global_step‚7749 %loss/total_loss‚41.37571375043427 $ _timestamp‚1.788444108427325e+09 1loss/FBt4/assignment_loss‚0.06186092823433379 /loss/FRt2/detection_loss‚2.4043074978722467 0loss/FRt1/assignment_loss‚1.2971999371051788 0loss/FRt4/assignment_loss‚2.8223484739661218 2loss/SRqqt‚"Ø¥11/assignment_loss‚0.8716233101706686 3loss/FBt3/assignment_loss‚0.0016921018541324884 1loss/FBt2/assignment_loss‚0.05639138499585291 /loss/FRt3/detection_loss‚2.3915921965497544 /loss/FRt4/detection_loss‚2.4983855463564395 1loss/SRqqt3/detection_loss‚2.5887282202320714 /loss/FBt2/detection_loss‚2.4541541039943695 /loss/FRt1/detection_loss‚1.5115625495836138 2loss/SRqqt3/assignment_loss‚1.5649402955126377 _step‚637 /loss/FRt3/assignment_loss‚3.394915042604719 1loss/SRqqt2/assignment_loss‚1.586247556758198ýr/—cŸ×‚(:” ÒûåÔæû¢É gpu.0.gpu‚64.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.598' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.519600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜íÌfCØ‚(:¸ ÒûåÔ¼åŒË disk.sda3.in‚0.5625# disk.dm-0.out‚58048.10986328125( proc.memory.percent‚16.0611629486084 network.sent‚ 1.928088e+07! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚58048.10986328125 disk.rbd2.out‚548.2412109375% disk./.usageGB‚1108.1934356689453 disk.sda3.out‚ 241.16015625 disk.rbd0.out‚0# memory_percent‚89.1465425491333) proc.memory.availableMB‚ 1778.23046875 network.recv‚ 2.378349e+06 disk.rbd3.out‚0& disk.nvme0n1.out‚58048.10986328125 disk.dm-0.in‚ 1754.82421875" proc.memory.rssMB‚ 2631.4609375 cpu‚25.321021420465343 proc.cpu.threads‚17 disk.rbd1.out‚ 3.2470703125) disk./.usagePercent‚74.37179130396885 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 241.16015625 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 241.16015625 G] ”Ù‚(:‰ áûåÔ©ó“É gpu.0.gpu‚71.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚262.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ú:áCÚ‚(:¸ áûåÔ¡¿ Ë disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚58145.98876953125 disk.dm-0.in‚ 1754.82421875$ memory_percent‚89.14449214935303& disk.nvme0n1.out‚58145.98876953125# disk.nvme0c0n1.in‚ 1754.82421875 disk.dm-1.out‚ 241.60546875 disk.rbd1.in‚0( proc.memory.percent‚16.0611629486084 network.recv‚ 2.385243e+06 disk.dm-1.in‚0.5625 disk.rbd3.out‚0 network.sent‚ 1.93903e+07 disk.rbd1.out‚ 3.2470703125 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1108.1934394836426! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1778.56640625 cpu‚25.320441841669965) disk./.usagePercent‚74.37179155997637 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 241.60546875# disk.dm-0.out‚58145.98876953125 disk.drbd1777.in‚0 disk.rbd2.out‚549.7763671875 disk.sda.out‚ 241.60546875" proc.memory.rssMB‚ 2631.4609375 proc.cpu.threads‚17ÄŒ7”Û‚(:‰ ðûåÔ¾©¦É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚256.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡®FÜ‚(:» ðûåÔ‚–˜Ë network.sent‚ 1.9396119e+07 network.recv‚ 2.389594e+06 disk.sda1.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 1778.56640625 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.31005859375 disk.rbd2.in‚0 disk.rbd2.out‚551.4287109375& disk.nvme0n1.out‚58146.25439453125 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.82421875 disk.dm-1.in‚0.5625) disk./.usagePercent‚74.37179155997637% disk./.usageGB‚1108.1934394836426 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 242.18359375 disk.drbd1777.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚58146.25439453125 disk.drbd1777.in‚0$ memory_percent‚89.14449214935303( proc.memory.percent‚16.0611629486084 disk.sda3.out‚ 242.18359375 disk.sda.in‚0.5625 disk.sda.out‚ 242.18359375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1754.82421875( disk.nvme0c0n1.out‚58146.25439453125" proc.memory.rssMB‚ 2631.4609375 disk.rbd0.out‚0 cpu‚25.319791026767682¸´áE2Ý- _wandbruntime‚6586  _runtime‚6586^x©ÈÞ ' lr-AdamW/pg1‚0.00013062567562915892 ' lr-AdamW/pg2‚0.00013062567562915892 trainer/global_step‚7799 $ _timestamp‚1.788444146551923e+09 _runtime‚6586.596648678 _step‚638ı±/ÍßÇ ' lr-AdamW/pg1‚0.00013062567562915892 ' lr-AdamW/pg2‚0.00013062567562915892 trainer/global_step‚7799 $ _timestamp‚1.788444146551923e+09 _runtime‚6586.596648678 _step‚638þü"I2à- _wandbruntime‚6586  _runtime‚6586Š“•E(áŠP QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145501119 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚884320839052288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚869846007287808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19827752593408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260647368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚870403472497904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚744111187507200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14450836428396 trainer/global_step‚7799 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14474831764480 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115166289 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚763938911966208 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115166283 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884441465526536e+09 _runtime‚6586.597517234 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚869846007287808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚855955234354176 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145501119 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚763938984399872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14472231555072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚884320839052288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128815234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚884318212451328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14474831764480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚884318212451328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14472231555072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145481085 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54086087 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115166289 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _step‚639 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚869845980896256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚744111159372800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115166283 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚870406096210540 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74728686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74728683 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145501119 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚763938984399872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14472231555072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚884320839052288 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128815234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚884318212451328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14474831764480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚884318212451328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14472231555072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145481085 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54086087 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115166289 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _step‚639 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚869845980896256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚744111159372800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115166283 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚870406096210540 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74728686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74728683 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚764033901417472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884441475148938e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145499319 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145519379 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 _runtime‚6587.560102695 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145519379 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14476664106496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74737882 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14450067486396 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚870511643014844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884427984485376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚764033826899968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7799 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚884430611099648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128830634 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14474063883776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚856061575528448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128831075 _step‚640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74737879 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115180525 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14452665771272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679838 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚856061600956416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚870514266727688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884430611099648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚869953946993152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679838 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115180525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441`Ú:'å™N PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128830634 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14474063883776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚856061575528448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128831075 _step‚640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74737879 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115180525 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14452665771272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679838 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚856061600956416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚870514266727688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884430611099648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚869953946993152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679838 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115180525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚744203575560704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚764033901417472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884441475148938e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145499319 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145519379 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 _runtime‚6587.560102695 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145519379 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14476664106496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74737882 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14450067486396 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚870511643014844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884427984485376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚764033826899968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7799 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚884430611099648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456€ÄÇNÎ2æ- _wandbruntime‚6587  _runtime‚6587èrZ¸RçÌ /loss/FRt4/assignment_loss‚2.587506629526615 2loss/SRqqt2/assignment_loss‚1.3099647331252402 /loss/FBt2/assignment_loss‚0.318486132635735 /loss/SRqqt1/detection_loss‚1.10112069202587 1loss/SRqqt2/detection_loss‚1.7194671202816216 /loss/FRt2/assignment_loss‚2.217777746788999 /loss/FBt3/assignment_loss‚0.624820926511893 .loss/FRt3/detection_loss‚2.506861618479184 /loss/FRt4/detection_loss‚2.3194711478427052 0loss/SRqqt4/detection_loss‚2.002682964612798 epoch‚6 _step‚641 /loss/FRt2/detection_loss‚2.2471830920970186 .loss/FBt4/detection_loss‚1.953500397503376 1loss/FBt1/assignment_loss‚0.12739144046395434 0loss/SRqqt3/detection_loss‚2.345338886275011 %loss/total_loss‚39.10549117434931 $ _timestamp‚1.788444147516656e+09 _runtime‚6587.561229207 2loss/SRqqt4/assignment_loss‚1.1927129734313207 /loss/FRt3/assignment_loss‚3.605006350624946 2loss/SRqqt3/assignment_loss‚1.3390249785254984 /loss/FBt3/detection_loss‚2.3940060257911684 2loss/SRqqt1/assignment_loss‚0.8767448461614549 .loss/FRt1/detection_loss‚1.630685537915539 /loss/FBt1/detection_loss‚0.6986085784757573 .loss/FBt2/detection_loss‚2.413780629634857 trainer/global_step‚7799 0loss/FRt1/assignment_loss‚1.3511323310710766 0loss/FBt4/assignment_loss‚0.2222153945476748K*O©WèÑ 1loss/FBt1/assignment_loss‚0.12739144046395434 0loss/SRqqt3/detection_loss‚2.345338886275011 %loss/total_loss‚39.10549117434931 $ _timestamp‚1.788444147516656e+09 _runtime‚6587.561229207 2loss/SRqqt4/assignment_loss‚1.1927129734313207 /loss/FRt3/assignment_loss‚3.605006350624946 2loss/SRqqt3/assignment_loss‚1.3390249785254984 /loss/FBt3/detection_loss‚2.3940060257911684 2loss/SRqqt1/assignment_loss‚0.8767448461614549 .loss/FRt1/detection_loss‚1.630685537915539 /loss/FBt1/detection_loss‚0.6986085784757573 .loss/FBt2/detection_loss‚2.413780629634857 trainer/global_step‚7799 0loss/FRt1/assignment_loss‚1.3511323310710766 0loss/FBt4/assignment_loss‚0.2222153945476748 /loss/FRt4/assignment_loss‚2.587506629526615 2loss/SRqqt2/assignment_loss‚1.3099647331252402 /loss/FBt2/assignment_loss‚0.318486132635735 /loss/SRqqt1/detection_loss‚1.10112069202587 1loss/SRqqt2/detection_loss‚1.7194671202816216 /loss/FRt2/assignment_loss‚2.217777746788999 /loss/FBt3/assignment_loss‚0.624820926511893 .loss/FRt3/detection_loss‚2.506861618479184 /loss/FRt4/detection_loss‚2.3194711478427052 0loss/SRqqt4/detection_loss‚2.002682964612798 epoch‚6 _step‚641 /loss/FRt2/detection_loss‚2.2471830920970186 .loss/FBt4/detection_loss‚1.953500397503376O«ûŸé‚(:” ÿûåÔÁ­­É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚159.557' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.911399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wζÿ1ê‚(:¦ ÿûåÔ¹çË disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda3.out‚242.625 disk.dm-1.out‚242.625( proc.memory.percent‚16.0611629486084' disk./.usagePercent‚74.371796168112 disk.sda.in‚0.5625 disk.rbd3.in‚0$ memory_percent‚89.14451599121094 cpu‚25.319180840992484 network.recv‚ 2.394143e+06 disk.drbd1777.out‚0 disk.rbd2.out‚551.4560546875 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875( disk.nvme0c0n1.out‚58244.08251953125# disk.dm-0.out‚58244.08251953125 disk.dm-1.in‚0.5625% disk./.usageGB‚1108.1935081481934 disk.rbd0.in‚0% proc.memory.availableMB‚ 1778.5625" proc.memory.rssMB‚ 2631.4609375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1754.82421875 disk.rbd1.out‚ 3.31005859375 disk.sda.out‚242.625 disk.rbd3.out‚0& disk.nvme0n1.out‚58244.08251953125 network.sent‚ 1.9402421e+07 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.82421875©)C“ë‚(:ˆ ŽüåÔÇü¡É gpu.0.gpu‚75.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚271.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&4Ý>ì‚(:³ ŽüåÔè˳Ë disk.sda.out‚ 242.9765625! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.in‚0.5625 proc.cpu.threads‚17) disk./.usagePercent‚74.37179232799899% disk./.usageGB‚1108.1934509277344 disk.rbd2.out‚553.1083984375& disk.nvme0n1.out‚58244.39892578125 disk.rbd1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.rbd1.out‚ 3.373046875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 242.9765625 cpu‚25.31857072313702 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚58244.39892578125 disk.dm-0.in‚ 1754.82421875 network.sent‚ 1.9511393e+07 network.recv‚ 2.400288e+06 disk.rbd0.in‚0 disk.dm-1.in‚0.5625( proc.memory.percent‚16.0611629486084 disk.sda.in‚0.5625 disk.rbd0.out‚0 disk.drbd1777.out‚0' proc.memory.availableMB‚ 1778.671875" proc.memory.rssMB‚ 2631.4609375# disk.dm-0.out‚58244.39892578125 disk.dm-1.out‚ 242.9765625 disk.drbd1777.in‚0$ memory_percent‚89.14384841918945–92í- _wandbruntime‚6625  _runtime‚6625¤ íÁÉîà ' lr-AdamW/pg2‚0.00012798788205740523 trainer/global_step‚7849 % _timestamp‚1.7884441855403802e+09 _runtime‚6625.585017305 _step‚642 ' lr-AdamW/pg1‚0.00012798788205740523êé7ÆÎïÈ ' lr-AdamW/pg1‚0.00012798788205740523 ' lr-AdamW/pg2‚0.00012798788205740523 trainer/global_step‚7849 % _timestamp‚1.7884441855403802e+09 _runtime‚6625.585017305 _step‚642‚iäç2ð- _wandbruntime‚6625  _runtime‚6625ôs,A(ñ‰P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294260 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146396137 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768683941568000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚889809455168512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54421344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861272318495744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6625.585991481 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚889809455168512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146396137 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚875242992555008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129609827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚875242966163456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚889806828567552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129610304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚861272293067776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115878089 trainer/global_step‚7849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115878083 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115878083 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚643 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875814635648748 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14566462613504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575'zU[ òŽP NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294260 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146396137 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768683941568000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚889809455168512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54421344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861272318495744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6625.585991481 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚889809455168512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146396137 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚875242992555008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129609827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚875242966163456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚889806828567552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129610304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚861272293067776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115878089 trainer/global_step‚7849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115878083 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115878083 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚643 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875814635648748 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14566462613504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54428056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14565694732800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚861378659670016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875350905868800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262306690 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14565694732800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14541548211004 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚644 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748822978235904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884441862767308e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _runtime‚ 6626.32188895 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19955854512640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262306690 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14568294955520 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129626202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875920182453052 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768778832748544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875350932260352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115892319 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129625735 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146434431 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146414371 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875350905868800 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115892325 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115892325 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146414371 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚889919227215872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚889916600601600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14544146495880 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14568294955520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146434431 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768778756133888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚889919227215872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401ƒ³'õ˜N =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262306690 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14568294955520 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129626202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875920182453052 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768778832748544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875350932260352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115892319 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129625735 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146434431 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146414371 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875350905868800 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115892325 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115892325 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146414371 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚889919227215872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚889916600601600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14544146495880 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14568294955520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146434431 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768778756133888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚889919227215872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115892319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54428520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19955806032384 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54428056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14565694732800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚861378659670016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875350905868800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262306690 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14565694732800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14541548211004 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚644 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748822978235904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884441862767308e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _runtime‚ 6626.32188895 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19955854512640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0„ñãvD2ö- _wandbruntime‚6626  _runtime‚6626d2uí]÷× 0loss/FRt1/assignment_loss‚1.1149173128989436 1loss/SRqqt4/detection_loss‚1.7872199778463327 /loss/FBt2/detection_loss‚2.1911987721920014 trainer/global_step‚7849 0loss/FBt3/assignment_loss‚0.6165817226089226 .loss/FRt4/detection_loss‚2.477943180128932 /loss/FRt4/assignment_loss‚2.798807729035616 2loss/FBt4/assignment_loss‚0.007293331390246749 /loss/FRt3/detection_loss‚2.2955271175321306 &loss/total_loss‚38.481447100542034 1loss/SRqqt2/detection_loss‚1.9514767511188984 0loss/SRqqt3/detection_loss‚2.347398736647197 0loss/FBt1/detection_loss‚0.45342880506068467 2loss/SRqqt2/assignment_loss‚1.2762371238321066 0loss/SRqqt3/assignment_loss‚1.13728791815894 /loss/FBt4/detection_loss‚2.5714923977851867 $ _timestamp‚1.788444186278528e+09 0loss/FRt2/assignment_loss‚2.8104024637829172 1loss/SRqqt4/assignment_loss‚1.491138701564541 1loss/FBt1/assignment_loss‚0.11011439286255868 0loss/FRt3/assignment_loss‚2.8803604790504944 2loss/SRqqt1/assignment_loss‚0.8185617788071731 1loss/FBt2/assignment_loss‚0.04985618849750608 epoch‚6 _runtime‚6626.323111965 _step‚645 /loss/FRt1/detection_loss‚1.5925632601063098 /loss/FRt2/detection_loss‚2.3843363042582166 1loss/SRqqt1/detection_loss‚0.8693414143693279 .loss/FBt3/detection_loss‚2.447961241006851ýùbøÜ /loss/FRt1/detection_loss‚1.5925632601063098 /loss/FRt2/detection_loss‚2.3843363042582166 1loss/SRqqt1/detection_loss‚0.8693414143693279 .loss/FBt3/detection_loss‚2.447961241006851 0loss/FRt1/assignment_loss‚1.1149173128989436 1loss/SRqqt4/detection_loss‚1.7872199778463327 /loss/FBt2/detection_loss‚2.1911987721920014 trainer/global_step‚7849 0loss/FBt3/assignment_loss‚0.6165817226089226 .loss/FRt4/detection_loss‚2.477943180128932 /loss/FRt4/assignment_loss‚2.798807729035616 2loss/FBt4/assignment_loss‚0.007293331390246749 /loss/FRt3/detection_loss‚2.2955271175321306 &loss/total_loss‚38.481447100542034 1loss/SRqqt2/detection_loss‚1.9514767511188984 0loss/SRqqt3/detection_loss‚2.347398736647197 0loss/FBt1/detection_loss‚0.45342880506068467 2loss/SRqqt2/assignment_loss‚1.2762371238321066 0loss/SRqqt3/assignment_loss‚1.13728791815894 /loss/FBt4/detection_loss‚2.5714923977851867 $ _timestamp‚1.788444186278528e+09 0loss/FRt2/assignment_loss‚2.8104024637829172 1loss/SRqqt4/assignment_loss‚1.491138701564541 1loss/FBt1/assignment_loss‚0.11011439286255868 0loss/FRt3/assignment_loss‚2.8803604790504944 2loss/SRqqt1/assignment_loss‚0.8185617788071731 1loss/FBt2/assignment_loss‚0.04985618849750608 epoch‚6 _runtime‚6626.323111965 _step‚645…©$žù‚(:“ üåÔ«Š©É gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚264.119' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.82380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Íh-ú‚(:¢ üåÔæò—Ë disk.sda1.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚58244.8212890625' disk.nvme0c0n1.out‚58244.8212890625( proc.memory.percent‚16.0611629486084% disk./.usageGB‚1108.1932678222656 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0 disk.sda3.in‚0.5625" disk.dm-0.out‚58244.8212890625# memory_percent‚89.1439437866211# disk.nvme0c0n1.in‚ 1754.82421875 network.recv‚ 2.404556e+06 disk.rbd2.out‚553.1357421875 disk.rbd3.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 1778.65625 disk.sda.out‚243.625) disk./.usagePercent‚74.37178003963734! disk.nvme0n1.in‚ 1754.82421875 disk.sda.in‚0.5625 network.sent‚ 1.9517489e+07 disk.sda2.in‚0 disk.dm-0.in‚ 1754.82421875 cpu‚25.31792971204528 disk.dm-1.out‚243.625 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚243.625" proc.memory.rssMB‚ 2631.4609375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0ªœžû‚(:“ ¬üåÔº˜…É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚281.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D[H¯7ü‚(:¯ ¬üåÔÓ²ÓÊ disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.out‚0 disk.dm-1.out‚ 243.984375( proc.memory.percent‚16.0611629486084% disk./.usageGB‚1108.1932411193848' disk.nvme0c0n1.out‚58342.3798828125 disk.drbd1777.in‚0 disk.sda.out‚ 243.984375% Ö70disk.nvme0n1.out‚58342.3798828125 disk.sda3.out‚ 243.984375 proc.cpu.threads‚17 network.sent‚ 1.9626707e+07# disk.nvme0c0n1.in‚ 1754.82421875 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.rbd3.in‚0 network.recv‚ 2.411344e+06 disk.dm-0.in‚ 1754.82421875 disk.rbd1.out‚ 3.373046875! disk.rbd2.out‚554.85888671875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0.5625$ memory_percent‚89.14484977722168 disk.rbd0.in‚0" disk.dm-0.out‚58342.3798828125 disk.dm-1.in‚0.5625 disk.sda2.in‚0( proc.memory.availableMB‚ 1778.5078125" proc.memory.rssMB‚ 2631.4609375 cpu‚25.317294124649003( disk./.usagePercent‚74.3717782475846 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0BÀL”ý‚(:‰ »üåÔÂõ¡É gpu.0.gpu‚58.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05e®?þ‚(:´ »üåÔ›ó¯Ë disk.drbd1777.out‚0! disk.rbd2.out‚556.39404296875 disk.sda.out‚ 244.546875 disk.rbd0.in‚0( proc.memory.percent‚16.0611629486084 disk.rbd1.out‚ 3.373046875 disk.dm-0.in‚ 1754.82421875 network.recv‚ 2.415814e+06" proc.memory.rssMB‚ 2631.4609375 proc.cpu.threads‚17 disk.rbd3.in‚0& disk.nvme0n1.out‚58343.04150390625 disk.sda3.out‚ 244.546875# disk.nvme0c0n1.in‚ 1754.82421875$ memory_percent‚89.14334774017334 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 network.sent‚ 1.9633078e+07 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 244.546875) proc.memory.availableMB‚ 1778.75390625) disk./.usagePercent‚74.37178208769761 disk.sda.in‚0.5625 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚58343.04150390625# disk.dm-0.out‚58343.04150390625 cpu‚25.316726198928247% disk./.usageGB‚1108.1932983398438 disk.drbd1777.in‚0 disk.sda1.out‚0ãwœ82ÿ- _wandbruntime‚6664  _runtime‚6664žävÉ€ à trainer/global_step‚7899 % _timestamp‚1.7884442245762603e+09 _runtime‚6664.620891818 _step‚646 ' lr-AdamW/pg1‚0.00012536379468105395 ' lr-AdamW/pg2‚0.000125363794681053958OÔGÎ È ' lr-AdamW/pg2‚0.00012536379468105395 trainer/global_step‚7899 % _timestamp‚1.7884442245762603e+09 _runtime‚6664.620891818 _step‚646 ' lr-AdamW/pg1‚0.00012536379468105395†ã?ã 2‚ - _wandbruntime‚6664  _runtime‚6664 éÔÙ(ƒ ŠP H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75648286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚895298071284736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147331223 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14658093462528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14655493253120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263901072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880639951430656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116589883 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75648283 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880639977822208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚753349992857600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20078754142720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14633797877612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880639977822208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147311189 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚753349964723200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚895298071284736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54757734 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚866589377209344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14631199592944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880639951430656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚773428795493888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54758222 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚895295444683776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14658093462528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14655493253120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20078802636288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚866589351781376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚895295444683776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14658093462528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14655493253120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20078802636288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚866589351781376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚895298071284736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147331223 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14658093462528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14655493253120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263901072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880639951430656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116589883 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75648283 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880639977822208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚753349992857600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20078754142720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14633797877612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880639977822208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147311189 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚753349964723200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚895298071284736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54757734 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚866589377209344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14631199592944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880639951430656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚773428795493888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54758222 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130421814 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚895407843332096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚866695718383616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54764786 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20081259125760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14633028935612 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14659925804544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880747917527552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚866695692955648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚880747917527552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147329423 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚753442352776704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130422268 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚880747891136000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14657325581824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚881331345604104 _step‚648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚895405216717824 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 trainer/global_step‚7899 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884442253096285e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚880747891136000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147349³—¯483 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147329423 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20081307606016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116604125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0<Ãi¸'‡ ™N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116604125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6665.354765612 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14657325581824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75657482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130421814 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚895407843332096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚866695718383616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54764786 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20081259125760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14633028935612 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14659925804544 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880747917527552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚866695692955648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚880747917527552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147329423 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚753442352776704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130422268 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚880747891136000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14657325581824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚881331345604104 _step‚648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚895405216717824 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 trainer/global_step‚7899 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884442253096285e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚880747891136000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147349483 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147329423 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20081307606016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720ˆJ¥}2ˆ - _wandbruntime‚6665  _runtime‚6665ôúü\‰ Ö 2loss/SRqqt1/assignment_loss‚1.2371026454875094 0loss/FRt3/assignment_loss‚3.6228556489944457 2loss/SRqqt4/assignment_loss‚1.3760442236810924 1loss/SRqqt1/detection_loss‚1.0543757220882861 /loss/FBt3/detection_loss‚2.9788344353437424 _step‚649 %loss/total_loss‚40.30510901910903 0loss/FRt4/assignment_loss‚2.8086345619625517 1loss/SRqqt2/assignment_loss‚1.501248133130545 1loss/FBt1/assignment_loss‚0.11260787557725348 .loss/FRt4/detection_loss‚2.365453033314811 .loss/FBt2/detection_loss‚1.706517646710078 1loss/FBt2/assignment_loss‚0.14750119998158576 0loss/FBt4/assignment_loss‚0.4156864302698523 0loss/FBt1/detection_loss‚0.37750785096338196 1loss/SRqqt4/detection_loss‚1.7154645644128323 trainer/global_step‚7899 /loss/FRt2/assignment_loss‚2.426927987824787 2loss/SRqqt3/assignment_loss‚1.7178740976378322 .loss/FRt1/detection_loss‚1.580454036680072 0loss/SRqqt3/detection_loss‚2.127485765144229 _runtime‚6665.356079278 /loss/FBt4/detection_loss‚2.3548773288726808 epoch‚6 0loss/FRt1/assignment_loss‚1.1669320824534395 .loss/FRt2/detection_loss‚2.575206046077338 /loss/FRt3/detection_loss‚2.9136809965310384 1loss/FBt3/assignment_loss‚0.15812566247041104 1loss/SRqqt2/detection_loss‚1.8637110434992368 % _timestamp‚1.7884442253115082e+09{fèWaŠ Û 0loss/FRt3/assignment_loss‚3.6228556489944457 2loss/SRqqt4/assignment_loss‚1.3760442236810924 1loss/SRqqt1/detection_loss‚1.0543757220882861 /loss/FBt3/detection_loss‚2.9788344353437424 _step‚649 %loss/total_loss‚40.30510901910903 0loss/FRt4/assignment_loss‚2.8086345619625517 1loss/SRqqt2/assignment_loss‚1.501248133130545 1loss/FBt1/assignment_loss‚0.11260787557725348 .loss/FRt4/detection_loss‚2.365453033314811 .loss/FBt2/detection_loss‚1.706517646710078 1loss/FBt2/assignment_loss‚0.14750119998158576 0loss/FBt4/assignment_loss‚0.4156864302698523 0loss/FBt1/detection_loss‚0.37750785096338196 1loss/SRqqt4/detection_loss‚1.7154645644128323 trainer/global_step‚7899 /loss/FRt2/assignment_loss‚2.426927987824787 2loss/SRqqt3/assignment_loss‚1.7178740976378322 .loss/FRt1/detection_loss‚1.580454036680072 0loss/SRqqt3/detection_loss‚2.127485765144229 _runtime‚6665.356079278 /loss/FBt4/detection_loss‚2.3548773288726808 epoch‚6 0loss/FRt1/assignment_loss‚1.1669320824534395 .loss/FRt2/detection_loss‚2.575206046077338 /loss/FRt3/detection_loss‚2.9136809965310384 1loss/FBt3/assignment_loss‚0.15812566247041104 1loss/SRqqt2/detection_loss‚1.8637110434992368 % _timestamp‚1.7884442253115082e+09 2loss/SRqqt1/assignment_loss‚1.2371026454875094‰»°Ô®Ÿ‹ ‚(:” ÊüåÔœî¢É gpu.0.gpu‚58.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚267.371' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.474199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æMKDŒ ‚(:¹ ÊüåÔ†éüË disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.82421875# disk.nvme0c0n1.in‚ 1754.82421875( disk.nvme0c0n1.out‚58440.59619140625$ memory_percent‚89.14480209350586 network.recv‚ 2.418596e+06 disk.sda.in‚0.5625 disk.sda3.in‚0.5625 disk.dm-1.out‚ 244.97265625# proc.memory.rssMB‚ 2632.20703125 cpu‚25.31612457639071% disk./.usageGB‚1108.1937828063965! disk.rbd2.out‚556.53857421875 disk.sda2.out‚0 disk.dm-0.in‚ 1754.82421875 proc.cpu.threads‚17 network.sent‚ 1.9637376e+07 disk.rbd2.in‚0 disk.sda.out‚ 244.97265625& disk.nvme0n1.out‚58440.59619140625 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 244.97265625 disk.rbd0.in‚0) proc.memory.percent‚16.06571674346924) disk./.usagePercent‚74.37181460065446 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚58440.59619140625' proc.memory.availableMB‚ 1778.515625 disk.rbd3.in‚0"µŸLž ‚(:“ ÙüåÔ‰ÞžÉ gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚49.0 gpu.0.powerWatts‚282.453' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.49059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝŽ‚·?Ž ‚(:´ ÙüåÔùò¼Ë network.recv‚ 2.425154e+06) disk./.usagePercent‚74.37179975221747 disk.sda1.out‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1778.37890625 disk.rbd3.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚16.06571674346924 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0% disk.nvme0n1.out‚58441.0595703125! disk.rbd2.out‚558.07373046875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2632.20703125 cpu‚25.31553409038746 disk.sda2.in‚0" disk.dm-0.out‚58441.0595703125 disk.dm-1.out‚ 245.6953125 proc.cpu.threads‚17 disk.drbd1777.in‚0 network.sent‚ 1.9746711e+07$ disk./.usageGB‚1108.193561553955 disk.sda.in‚0.5625 disk.sda.out‚ 245.6953125 disk.sda3.in‚0.5625 disk.sda3.out‚ 245.6953125' disk.nvme0c0n1.out‚58441.0595703125 disk.dm-0.in‚ 1754.82421875 disk.sda1.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.out‚0$ memory_percent‚89.14563655853271æT¼2 - _wandbruntime‚6703  _runtime‚6703ôÄ»UÈ Â _runtime‚ 6703.40779448 _step‚650 ' lr-AdamW/pg1‚0.00012275396035598118 ' lr-AdamW/pg2‚0.00012275396035598118 trainer/global_step‚7949 % _timestamp‚1.7884442633631325e+09·e²÷Í‘ Ç ' lr-AdamW/pg1‚0.00012275396035598118 ' lr-AdamW/pg2‚0.00012275396035598118 trainer/global_step‚7949 % _timestamp‚1.7884442633631325e+09 _runtime‚ 6703.40779448 _step‚650Š|ñZ2’ - _wandbruntime‚6703  _runtime‚6703Rš (“ ŠP LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148246275 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚900786687400960 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131202723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20204391939072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚886036936697856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚871906410494976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117301689 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7949 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚886036963089408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14747124102144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚474 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚886631714525164 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚6703.408773194 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76108086 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14747124102144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117301683 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚900784060800000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14749724311552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55094637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚900784060800000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117301683 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55094166 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚778173833868288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚900786687400960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚778173759337472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884442633638833e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚886036963089408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131202249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚886629090812528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265547964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 _step‚651 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚757969395532800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚871906435922944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117301689 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14725278602220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76108083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148226241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20204438335488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148226241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14722680317552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚757969367398400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚886036936697856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚900784060800000 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117301683 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55094166 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚778173833868288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚900786687400960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚778173759337472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884442633638833e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚886036963089408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131202249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚886629090812528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265547964 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚471 _step‚651 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚757969395532800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527924 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚871906435922944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117301689 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14725278602220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76108083 \ODeviceSXÅs`ätatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148226241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20204438335488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148226241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14722680317552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚757969367398400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚886036936697856 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚886631714525164 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚6703.408773194 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76108086 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14747124102144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117301683 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚900784060800000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14749724311552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55094637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚778268754463232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚872012751669248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14724509660220 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117315919 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131218132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148244475 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14751556653568 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚886739885042312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55100853 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚886737261329468 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚900896459448320 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117315925 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚886144876403200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14751556653568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14748956430848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20206970876928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚7949 _step‚652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚900896459448320 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14727107945096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758061783586304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚886144902794752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148244475 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117315925 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884442641011508e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚886144902794752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560394 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55101289 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚886144876403200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131218571 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76117279 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚6704.146219006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14748956430848 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148264535 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401É™û'— ™N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117315925 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 % _timestamp‚1.7884442641011508e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚886144902794752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560394 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55101289 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚886144876403200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131218571 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76117279 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚6704.146219006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14748956430848 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148264535 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚778268754463232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚872012751669248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14724509660220 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117315919 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131218132 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148244475 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14751556653568 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚886739885042312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55100853 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚886737261329468 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚900896459448320 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117315925 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚886144876403200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14751556653568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14748956430848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20206970876928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚7949 _step‚652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚900896459448320 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14727107945096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758061783586304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚886144902794752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148244475ŒPz÷A2˜ - _wandbruntime‚6704  _runtime‚6704÷1Wü[™ Õ .loss/FRt3/detection_loss‚2.605384775137985 1loss/SRqqt4/detection_loss‚1.7788494847169736 .loss/FBt2/detection_loss‚2.675261974334717 &loss/total_loss‚39.259841324242515 /loss/FRt2/detection_loss‚2.1877561027805013 /loss/SRqqt2/detection_loss‚1.89985989369452 .loss/FBt3/detection_loss‚2.604842185974121 0loss/FRt2/assignment_loss‚2.7071113884449005 0loss/FRt4/assignment_loss‚2.2242267529169717 1loss/FBt1/assignment_loss‚0.13000437349482644 1loss/SRqqt1/detection_loss‚0.9014679543016588 epoch‚6 % _timestamp‚1.7884442641029968e+09 0loss/FRt1/assignment_loss‚1.0930515396595002 2loss/SRqqt2/assignment_loss‚1.5208932637237011 2loss/SRqqt3/assignment_loss‚1.6437784338163004 _runtime‚6704.147584409 1loss/SRqqt1/assignment_loss‚0.832569887765738 1loss/FBt3/assignment_loss‚0.14345969332862296 1loss/FBt4/assignment_loss‚0.38775161482460263 /loss/FBt4/detection_loss‚2.2370237708091736 trainer/global_step‚7949 .loss/FRt3/assignment_loss‚3.38249122351408 1loss/SRqqt4/assignment_loss‚1.105023521021372 1loss/FBt2/assignment_loss‚0.36256942200790426 .loss/FRt4/detection_loss‚2.436083818806542 1loss/SRqqt3/detection_loss‚2.2616100940439434 /loss/FRt1/detection_loss‚1.6526493818312884 0loss/FBt1/detection_loss‚0.48612077329256626 _step‚6539«¸`š Ú epoch‚6 % _timestamp‚1.7884442641029968e+09 0loss/FRt1/assignment_loss‚1.0930515396595002 2loss/SRqqt2/assignment_loss‚1.5208932637237011 2loss/SRqqt3/assignment_loss‚1.6437784338163004 _runtime‚6704.147584409 1loss/SRqqt1/assignment_loss‚0.832569887765738 1loss/FBt3/assignment_loss‚0.14345969332862296 1loss/FBt4/assignment_loss‚0.38775161482460263 /loss/FBt4/detection_loss‚2.2370237708091736 trainer/global_step‚7949 .loss/FRt3/assignment_loss‚3.38249122351408 1loss/SRqqt4/assignment_loss‚1.105023521021372 1loss/FBt2/assignment_loss‚0.36256942200790426 .loss/FRt4/detection_loss‚2.436083818806542 1loss/SRqqt3/detection_loss‚2.2616100940439434 /loss/FRt1/detection_loss‚1.6526493818312884 0loss/FBt1/detection_loss‚0.48612077329256626 _step‚653 .loss/FRt3/detection_loss‚2.605384775137985 1loss/SRqqt4/detection_loss‚1.7788494847169736 .loss/FBt2/detection_loss‚2.675261974334717 &loss/total_loss‚39.259841324242515 /loss/FRt2/detection_loss‚2.1877561027805013 /loss/SRqqt2/detection_loss‚1.89985989369452 .loss/FBt3/detection_loss‚2.604842185974121 0loss/FRt2/assignment_loss‚2.7071113884449005 0loss/FRt4/assignment_loss‚2.2242267529169717 1loss/FBt1/assignment_loss‚0.13000437349482644 1loss/SRqqt1/detection_loss‚0.9014679543016588žúü»“› ‚(:ˆ èüåÔý¤ŠÉ gpu.0.gpu‚59.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚95.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áøé<œ ‚(:± èüåÔÇÚìË disk.drbd1777.out‚0 network.recv‚ 2.429696e+06 disk.rbd3.out‚0 disk.sda3.out‚ 246.07421875 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875 disk.dm-0.in‚ 1754.82421875! disk.dm-0.out‚58441.388671875) proc.memory.availableMB‚ 1778.12890625 cpu‚25.31490252235684 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.in‚0 network.sent‚ 1.9752592e+07) disk./.usagePercent‚74.37186887425172 disk.rbd2.in‚0$ memory_percent‚89.14716243743896 disk.rbd1.out‚ 3.373046875# proc.memory.rssMB‚ 2632.20703125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.out‚ 246.07421875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚58441.388671875 disk.rbd2.out‚ 558.2265625% disk./.usageGB‚1108.1945915222168! disk.nvme0n1.in‚ 1754.82421875$ disk.nvme0n1.out‚58441.388671875 disk.rbd0.out‚0 disk.dm-1.out‚ 246.07421875) proc.memory.percent‚16.06571674346924 disk.rbd3.in‚0 disk.sda1.out‚0Nòƒx” ‚(:‰ ÷üåÔŠÏ†É gpu.0.gpu‚59.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚267.061' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4122 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù&&ùAž ‚(:¶ ÷üåÔžó Ì# disk.dm-0.out‚58539.13134765625 disk.rbd1.in‚0 disk.rbd2.out‚ 559.73828125 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.dm-1.out‚ 246.4921875 disk.sda.out‚ 246.4921875 disk.sda1.out‚0 disk.sda3.in‚0.5625# proc.memory.rssMB‚ 2632.20703125 cpu‚25.314310341288312) disk./.usagePercent‚74.37186759421405( disk.nvme0c0n1.out‚58539.13134765625 disk.drbd1777.out‚0! disk.nvme0n1.in‚ 1754.82421875$ memory_percent‚89.14816379547119) proc.memory.availableMB‚ 1777.96484375 network.recv‚ 2.436556e+06 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.rbd3.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0) proc.memory.percent‚16.06571674346924 proc.cpu.threads‚17 network.sent‚ 1.9862301e+07 disk.rbd1.out‚ 3.373046875& disk.nvme0n1.out‚58539.13134765625 disk.sda3.out‚ 246.4921875% disk./.usageGB‚1108.1945724487305 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1754.82421875Rîô”Ÿ ‚(:‰ †ýåÔ¹ÂåÈ gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚237.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.4678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôØm@  ‚(:µ †ýåÔˆÂ–Ë disk.rbd0.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1754.82421875 cpu‚25.31369541937667 proc.cpu.threads‚17 disk.dm-1.out‚ 246.84765625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚58539.44775390625 disk.dm-0.in‚ 1754.82421875% disk./.usageGB‚1108.1945991516113 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0) proc.memory.percent‚16.06571674346924 disk.rbd2.out‚ 559.90625 disk.sda.out‚ 246.84765625 disk.sda3.out‚ 246.84765625$ memory_percent‚89.14673328399658# proc.memory.rssMB‚ 2632.20703125 network.sent‚ 1.9867551e+07& disk.nvme0n1.out‚58539.44775390625 disk.dm-1.in‚0.5625 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875# disk.dm-0.out‚58539.44775390625) proc.memory.availableMB‚ 1778.19921875 disk.rbd3.out‚0 network.recv‚ 2.440374e+06) disk./.usagePercent‚74.37186938626678 disk.rbd1.out‚ 3.373046875Ä£,2¡ - _wandbruntime‚6742  _runtime‚6742 iÉ¢ à ' lr-AdamW/pg2‚0.00012015892296774857 trainer/global_step‚7999 % _timestamp‚1.7884443023454285e+09 _runtime‚6742.390042079 _step‚654 ' lr-AdamW/pg1‚0.00012015892296774857yä­ Î£ È ' lr-AdamW/pg1‚0.00012015892296774857 ' lr-AdamW/pg2‚0.00012015892296774857 trainer/global_step‚7999 % _timestamp‚1.7884443023454285e+09 _runtime‚6742.390042079 _step‚654Ž±cÕu ¤ - _wa\G&&ndbruntime‚6742  _runtime‚6742tŸ[(¥ ŠP \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20330009515008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55432043 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚877223469208576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚906275303517184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚891433948356608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118013489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149141293 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚906275303517184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149141293 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚891433921965056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14838754951168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762588798208000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20330058008576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚906272676916224 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118013489 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132000394 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚762588770073600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚655 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚891433948356608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚891433921965056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76567886 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76567883 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚877223494636544 _runtime‚6742.390958777 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118013483 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131999926 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚782918779588608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884443023460987e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14841355160576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14841355160576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14814161042160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149161327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149161327 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚906272676916224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚782918856216576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55432508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14816759326828 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚892040253963372 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575ƒaB(¦ P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118013483 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55432043 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚877223469208576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚906275303517184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚891433948356608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118013489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149141293 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚906275303517184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149141293 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚891433921965056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14838754951168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762588798208000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20330058008576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚906272676916224 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118013489 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132000394 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚762588770073600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚655 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚891433948356608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚891433921965056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76567886 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76567883 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚877223494636544 _runtime‚6742.390958777 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118013483 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131999926 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚782918779588608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884443023460987e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14841355160576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14841355160576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14814161042160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149161327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149161327 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚906272676916224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚782918856216576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55432508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14816759326828 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚892040253963372 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20330009515008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132015694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118027725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 _runtime‚6743.128634525 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚783013679718400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20332567974400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132016148 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚877329810382848 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267187246 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14840587279872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884443030834854e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118027725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118027719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149179587 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚783013754235904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚877329835810816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚891541888061952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚892145800767676 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14818588669704 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14840587279872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14843187502592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0DT)q?© ™N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚892145800767676 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14818588669704 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14840587279872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14843187502592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118027719 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚892148424480520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55438615 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149159527 Fàe_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267207312 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14815990384828 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚906385075564544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132015694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118027725 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 _runtime‚6743.128634525 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚783013679718400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20332567974400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132016148 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚877329810382848 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267187246 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14840587279872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884443030834854e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118027725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118027719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149179587 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚783013754235904 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚877329835810816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚891541888061952~²2ÿ2ª - _wandbruntime‚6743  _runtime‚6743Ǽ™T« Î /loss/FRt3/assignment_loss‚3.461282992362976 2loss/SRqqt3/assignment_loss‚1.7033770457387736 _runtime‚6743.130014046 _step‚657 1loss/SRqqt4/assignment_loss‚1.012220971557227 0loss/SRqqt2/detection_loss‚1.764715090011939 .loss/FBt4/detection_loss‚3.256664538383484 epoch‚6 % _timestamp‚1.7884443030854423e+09 1loss/SRqqt1/detection_loss‚1.3960986434295775 1loss/SRqqt4/detection_loss‚1.4556570987809787 %loss/total_loss‚40.51296177667369 0loss/FBt1/assignment_loss‚0.2249961803265079 /loss/SRqqt3/detection_loss‚2.23075278982138 /loss/FBt2/detection_loss‚2.4082043983719568 .loss/FBt3/detection_loss‚3.185566985607147 /loss/FRt2/assignment_loss‚2.297505901048058 2loss/SRqqt2/assignment_loss‚1.0424638322721689 .loss/FRt3/detection_loss‚2.217796661425382 trainer/global_step‚7999 2loss/FBt4/assignment_loss‚0.041223585675470534 0loss/FRt1/assignment_loss‚1.6048530141512554 .loss/FBt2/assignment_loss‚0.25262236318932 .loss/FRt2/detection_loss‚2.678979040760743 /loss/FRt4/detection_loss‚2.2986291897567837 /loss/FBt1/detection_loss‚0.5170645282328636 /loss/FRt4/assignment_loss‚2.178467161276124 2loss/SRqqt1/assignment_loss‚0.9336641907691956 0loss/FBt3/assignment_loss‚0.0980672231700737 /loss/FRt1/detection_loss‚2.2520883505543074S ¤µY¬ Ó 2loss/SRqqt3/assignment_loss‚1.7033770457387736 _runtime‚6743.130014046 _step‚657 1loss/SRqqt4/assignment_loss‚1.012220971557227 0loss/SRqqt2/detection_loss‚1.764715090011939 .loss/FBt4/detection_loss‚3.256664538383484 epoch‚6 % _timestamp‚1.7884443030854423e+09 1loss/SRqqt1/detection_loss‚1.3960986434295775 1loss/SRqqt4/detection_loss‚1.4556570987809787 %loss/total_loss‚40.51296177667369 0loss/FBt1/assignment_loss‚0.2249961803265079 /loss/SRqqt3/detection_loss‚2.23075278982138 /loss/FBt2/detection_loss‚2.4082043983719568 .loss/FBt3/detection_loss‚3.185566985607147 /loss/FRt2/assignment_loss‚2.297505901048058 2loss/SRqqt2/assignment_loss‚1.0424638322721689 .loss/FRt3/detection_loss‚2.217796661425382 trainer/global_step‚7999 2loss/FBt4/assignment_loss‚0.041223585675470534 0loss/FRt1/assignment_loss‚1.6048530141512554 .loss/FBt2/assignment_loss‚0.25262236318932 .loss/FRt2/detection_loss‚2.678979040760743 /loss/FRt4/detection_loss‚2.2986291897567837 /loss/FBt1/detection_loss‚0.5170645282328636 /loss/FRt4/assignment_loss‚2.178467161276124 2loss/SRqqt1/assignment_loss‚0.9336641907691956 0loss/FBt3/assignment_loss‚0.0980672231700737 /loss/FRt1/detection_loss‚2.2520883505543074 /loss/FRt3/assignment_loss‚3.461282992362976‘4}Ž<”­ ‚(:‰ •ýåÔ¸‚ªÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í¶E,C® ‚(:¸ •ýåÔçÞËË$ disk./.usageGB‚1108.194580078125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.5625 proc.cpu.threads‚17 network.sent‚ 1.9873651e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 1754.82421875) proc.memory.percent‚16.06571674346924 network.recv‚ 2.444536e+06( disk.nvme0c0n1.out‚58637.10302734375) proc.memory.availableMB‚ 1778.10546875 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0.5625# proc.memory.rssMB‚ 2632.20703125 disk.rbd3.in‚0 disk.sda.out‚ 247.51953125 cpu‚25.313108482732375 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.dm-1.out‚ 247.51953125 disk.drbd1777.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚74.37186810622912 disk.sda2.out‚0 disk.sda3.out‚ 247.51953125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875 disk.rbd2.out‚ 561.41796875! disk.nvme0n1.in‚ 1754.82421875& disk.nvme0n1.out‚58637.10302734375# disk.nvme0c0n1.in‚ 1754.82421875# disk.dm-0.out‚58637.10302734375$ memory_percent‚89.14730548858643J‡!”¯ ‚(:‰ ¤ýåÔ»ð×È gpu.0.gpu‚58.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚268.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘%°’5° ‚(:ª ¤ýåÔŸ¸ÜË disk.rbd0.in‚0 cpu‚25.312526530659362 proc.cpu.threads‚17 network.sent‚ 1.9982798e+07 disk.sda1.in‚0 disk.drbd1777.out‚0$ memory_percent‚89.14661407470703 disk.rbd1.out‚ 3.373046875 disk.sda.in‚0.5625 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1754.82421875 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2632.20703125 disk.rbd1.in‚0 disk.rbd2.out‚ 561.51953125 disk.sda1.out‚0# disk.nvme0n1.out‚58637.50390625) proc.memory.percent‚16.06571674346924 network.recv‚ 2.451293e+06 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 248.0546875! disk.nvme0n1.in‚ 1754.82421875 disk.rbd0.out‚0 disk.dm-0.in‚ 1754.82421875) disk./.usagePercent‚74.37186836223665 disk.rbd3.in‚0& proc.memory.availableMB‚ 1778.21875 disk.sda.out‚ 248.0546875 disk.sda2.in‚0% disk.nvme0c0n1.out‚58637.50390625 disk.dm-0.out‚58637.50390625% disk./.usageGB‚1108.1945838928223 disk.rbd2.in‚0 disk.sda3.out‚ 248.0546875^‘Ÿ± ‚(:” ³ýåÔš•àÈ gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚143.098' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.619600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶o]L7² ‚(:¬ ³ýåÔ·®ÛË disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 248.546875( proc.memory.availableMB‚ 1778.1953125 disk.sda2.out‚0 disk.rbd0.in‚0) proc.memory.percent‚16.06571674346924 disk.sda.in‚0.5625 disk.rbd2.out‚ 563.03125# proc.memory.rssMB‚ 2632.20703125 proc.cpu.threads‚17 disk.sda.out‚ 248.546875% disk.nvme0n1.out‚58638.2822265625 disk.rbd2.in‚0$ memory_percent‚89.14675712585449 cpu‚25.311948046702227 network.recv‚ 2.455594e+06 disk.rbd3.out‚0 disk.dm-0.in‚ 1754.82421875' disk.nvme0c0n1.out‚58638.2822265625" disk.dm-0.out‚58638.2822265625 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875) disk./.usagePercent‚74.37187220234966 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.in‚0.5625% disk./.usageGB‚1108.1946411132812 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda3.out‚ 248.546875# disk.nvme0c0n1.in‚ 1754.82421875 network.sent‚ 1.9989194e+07oyLö2³ - _wandbruntime‚6781  _runtime‚6781ÃgÉ´ à _runtime‚6781.168667086 _step‚658 ' lr-AdamW/pg1‚0.00011757922331825827 ' lr-AdamW/pg2‚0.00011757922331825827 trainer/global_step‚8049 % _timestamp‚1.7884443411239161e+09A“íñε È ' lr-AdamW/pg2‚0.00011757922331825827 trainer/global_step‚8049 % _timestamp‚1.7884443411239161e+09 _runtime‚6781.168667086 _step‚658 ' lr-AdamW/pg1‚0.00011757922331825827’óàç)2¶ - _wandbruntime‚6781  _runtime‚67816ç&Ñ(· ‰P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118725283 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚911761293032448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14908240051436 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚911761293032448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14905641766768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118725289 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150056345 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882540527922176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150076379 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚896830907232256 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150076379 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6781.169590257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781628 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 _step‚659 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚911763919633408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20455240875008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20455289368576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896830933623808 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132794619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55767417 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14932986009600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 $ _timestamp‚1.788444341124695e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150056345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897448793401580 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚896830933623808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897446169688944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77027686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268801668 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132795103 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚787663490251776 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118725283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481Ÿ9Q¸ ŽP KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14930385800192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55766936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14932986009600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 $ _timestamp‚1.788444341124695e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150056345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897448793401580 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚896830933623808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897446169688944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77027686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268801668 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132795103 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚787663490251776 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118725283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118725283 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚911761293032448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14908240051436 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚911761293032448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14905641766768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118725289 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150056345 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882540527922176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150076379 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚896830907232256 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150076379 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6781.169590257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268781628 WGDeviceStatsMonitor.on_train_batch_start/allo& ."à cated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 _step‚659 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚911763919633408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20455240875008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20455289368576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896830933623808 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132794619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55767417 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132810654 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882646869096448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚767300588936704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 trainer/global_step‚8049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77036882 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20457787449344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897556963918728 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚6781.907133819 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150094639 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚911873691680768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚787758376386048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚767300560802304·{“²'» ™N EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚6781.907133819 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150094639 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚911873691680768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚787758376386048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚767300560802304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834164 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896938873329152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132811106 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896938846937600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14932218128896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚911873691680768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14910069394312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20457736871936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268814098 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118739525 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150094639 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884443418620014e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132810654 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882646869096448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚767300588936704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 trainer/global_step‚8049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77036882 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20457787449344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897556963918728”ŠÕÁM2¼ - _wandbruntime‚6781  _runtime‚6781u,@™V½ Ð /loss/FRt4/assignment_loss‚2.412338476431997 1loss/FBt1/assignment_loss‚0.09112105546193866 /loss/FBt1/detection_loss‚0.4878839875516646 /loss/FBt2/detection_loss‚3.0706170967647006 .loss/FBt3/detection_loss‚2.543494947751363 2loss/SRqqt1/assignment_loss‚0.8103527504426893 0loss/FBt3/assignment_loss‚0.1787675842526369 epoch‚6 0loss/FRt3/assignment_loss‚3.7530844397842884 1loss/FBt4/assignment_loss‚0.01797165074718318 0loss/SRqqt1/detection_loss‚0.753566331230104 trainer/global_step‚8049 _runtime‚6781.908615072 _step‚661 1loss/SRqqt2/detection_loss‚1.7949891767154138 /loss/FBt4/detection_loss‚2.6968211957386563 0loss/FRt1/assignment_loss‚1.1513757816580839 1loss/SRqqt3/assignment_loss‚1.556576480990962 0loss/FBt2/assignment_loss‚0.6508502786912557 /loss/FRt1/detection_loss‚1.6808860637247562 /loss/FRt3/detection_loss‚2.2227934093061776 -loss/FRt4/detection_loss‚2.41267533914039 0loss/SRqqt3/detection_loss‚2.155903072733628 .loss/FRt2/detection_loss‚2.598396882414818 0loss/FRt2/assignment_loss‚2.5845201937925246 1loss/SRqqt4/detection_loss‚1.6072196976414748 $loss/total_loss‚39.8019795173961 1loss/SRqqt2/assignment_loss‚1.203626684223612 2loss/SRqqt4/assignment_loss‚1.3661469402057784 $ _timestamp‚1.788444341864019e+09B[tS[¾ Õ /loss/FRt1/detection_loss‚1.6808860637247562 /loss/FRt3/detection_loss‚2.2227934093061776 -loss/FRt4/detection_loss‚2.41267533914039 0loss/SRqqt3/detection_loss‚2.155903072733628 .loss/FRt2/detection_loss‚2.598396882414818 0loss/FRt2/assignment_loss‚2.5845201937925246 1loss/SRqqt4/detection_loss‚1.6072196976414748 $loss/total_loss‚39.8019795173961 1loss/SRqqt2/assignment_loss‚1.203626684223612 2loss/SRqqt4/assignment_loss‚1.3661469402057784 $ _timestamp‚1.788444341864019e+09 /loss/FRt4/assignment_loss‚2.412338476431997 1loss/FBt1/assignment_loss‚0.09112105546193866 /loss/FBt1/detection_loss‚0.4878839875516646 /loss/FBt2/detection_loss‚3.0706170967647006 .loss/FBt3/detection_loss‚2.543494947751363 2loss/SRqqt1/assignment_loss‚0.8103527504426893 0loss/FBt3/assignment_loss‚0.1787675842526369 epoch‚6 0loss/FRt3/assignment_loss‚3.7530844397842884 1loss/FBt4/assignment_loss‚0.01797165074718318 0loss/SRqqt1/detection_loss‚0.753566331230104 trainer/global_step‚8049 _runtime‚6781.908615072 _step‚661 1loss/SRqqt2/detection_loss‚1.7949891767154138 /loss/FBt4/detection_loss‚2.6968211957386563 0loss/FRt1/assignment_loss‚1.1513757816580839 1loss/SRqqt3/assignment_loss‚1.556576480990962 0loss/FBt2/assignment_loss‚0.6508502786912557•°:£ÄŸ¿ ‚(:” ÂýåÔ›±ŒÉ gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚234.622' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.924400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™'ý:<À ‚(:± ÂýåÔÇò—Ë disk.sda3.out‚ 248.81640625 disk.dm-1.in‚0.5625 disk.rbd3.in‚0) proc.memory.percent‚16.06571674346924) disk./.usagePercent‚74.37183047312158 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.drbd1777.in‚0 disk.sda1.out‚0 network.recv‚ 2.45918e+06 disk.dm-0.in‚ 1754.82421875" disk.dm-0.out‚58735.7939453125 disk.rbd2.in‚0 disk.rbd2.out‚ 564.7109375 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.999433e+07 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1754.82421875# proc.memory.rssMB‚ 2632.20703125 cpu‚25.31137605871318 disk.sda.in‚0.5625 disk.sda.out‚ 248.81640625 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚89.14973735809326 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0) proc.memory.availableMB‚ 1777.70703125 disk.dm-1.out‚ 248.81640625$ disk./.usageGB‚1108.194019317627 disk.sda3.in‚0.5625' disk.nvme0c0n1.out‚58735.7939453125% disk.nvme0n1.out‚58735.7939453125›K‚¿Á ‚(:„ ÑýåÔð¿ˆÉ gpu.0.gpu‚68.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚99.3' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.86 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0䉾A ‚(:¶ ÑýåÔ¯˜ŽË disk.sda3.in‚0.5625 disk.drbd1777.in‚0 cpu‚25.31080323901191 proc.cpu.threads‚17 network.recv‚ 2.465951e+06% disk./.usageGB‚1108.1940002441406 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 564.7109375 disk.drbd1777.out‚0) disk./.usagePercent‚74.37182919308391# disk.nvme0c0n1.in‚ 1754.82421875 disk.dm-0.in‚ 1754.82421875# memory_percent‚89.1460657119751# proc.memory.rssMB‚ 2632.20703125 network.sent‚ 2.0103616e+07( disk.nvme0c0n1.out‚58736.22509765625 disk.dm-1.out‚ 249.46484375 disk.rbd1.out‚ 3.373046875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚58736.22509765625 disk.sda.in‚0.5625 disk.sda.out‚ 249.46484375 disk.sda2.out‚0 disk.dm-1.in‚0.5625) proc.memory.percent‚16.06571674346924! disk.nvme0n1.in‚ 1754.82421875 disk.sda3.out‚ 249.46484375 disk.rbd2.in‚0) proc.memory.availableMB‚ 1778.30859375 disk.sda1.out‚0& disk.nvme0n1.out‚58736.22509765625²Ôîu2à - _wandbruntime‚6820  _runtime‚6820š^OOÉÄ Ã ' lr-AdamW/pg1‚0.00011501539901305075 ' lr-AdamW/pg2‚0.00011501539901305075 trainer/global_step‚8099 % _timestamp‚1.7884443801842015e+09 _runtime‚6820.228874949 _step‚662ÿa~ÎÅ È _runtime‚6820.228874949 _step‚662 ' lr-AdamW/pg1‚0.00011501539901305075 ' lr-AdamW/pg2‚0.00011501539901305075 trainer/global_step‚8099 % _timestamp‚1.7884443801842015e+09–.MEª2Æ - _wandbruntime‚6820  _runtime‚6820Œ@¬Ù Ç ŠP M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150971397 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917249909148672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15024616858624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚792408533865984 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14999720776044 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 % _timestamp‚1.7884443801849241e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119437089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917252535749632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917249909148672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56105967 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚887857586635776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14997122491376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150971397 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚471 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚917252535749632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902227892499456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902227892499456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15022016649216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119437089 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20580881814016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚771827575424000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77487486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56105499 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚663 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8099 _runtime‚6820.229821433 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119437083 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119437083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150991431 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133593453 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133592982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15022016649216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚771827603558400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚887857612063744 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14999720776044 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 % _timestamp‚1.7884443801849241e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119437089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917252535749632 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917249909148672 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56105967 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚887857586635776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14997122491376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150971397 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚471 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚917252535749632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902227892499456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902227892499456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15022016649216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119437089 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20580881814016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚771827575424000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77487486 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56105499 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚663 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8099 _runtime‚6820.229821433 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119437083 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119437083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150991431 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133593453 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133592982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15022016649216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚771827603558400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚887857612063744 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270461016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56112513 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150989631 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151009691 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚887963953238016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119451325 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902335832204800 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119451325 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15023848977920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917359681182720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77496679 trainer/global_step‚8099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119451319 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15026449200640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133609195 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20583376943616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15001550118920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902335832204800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚902965503356936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚792503414938624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _step‚664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133608749 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20583423326720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133608749 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20583423326720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270461016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56112513 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150989631 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151009691 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚887963953238016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119451325 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902335832204800 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119451325 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15023848977920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917359681182720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77496679 trainer/global_step‚8099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119451319 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15026449200640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133609195 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20583376943616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15001550118920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902335832204800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚902965503356936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚792503414938624˜WxcÎ2Ì - _wandbruntime‚6820  _runtime‚6820]àÄáXÍ Ò 1loss/FBt3/assignment_loss‚0.18095530279388186 /loss/FBt4/detection_loss‚2.3926612794399262 trainer/global_step‚8099 /loss/FRt3/assignment_loss‚3.348271315296491 1loss/SRqqt1/assignment_loss‚0.785258673988874 2loss/SRqqt3/assignment_loss‚1.5958505370024894 1loss/SRqqt4/detection_loss‚1.7065003492214061 .loss/FBt2/detection_loss‚2.799139371285072 2loss/SRqqt4/assignment_loss‚1.5825270535079417 0loss/FBt1/assignment_loss‚0.1797649154058476 1loss/SRqqt1/detection_loss‚0.7714629253697773 1loss/SRqqt2/detection_loss‚1.8655631447380239 _runtime‚6820.966740711 &loss/total_loss‚39.271448182049056 /loss/FRt2/assignment_loss‚2.549296262652375 2loss/SRqqt2/assignment_loss‚1.3871129742315547 2loss/FBt2/assignment_loss‚0.014897083743282845 .loss/FRt2/detection_loss‚2.333020806312561 -loss/FBt3/detection_loss‚2.59521244764328 0loss/FRt4/assignment_loss‚3.0100572716106067 0loss/FBt4/assignment_loss‚0.6840788976331169 .loss/FRt1/detection_loss‚1.313333108376425 .loss/FBt1/detection_loss‚0.520511554215442 _step‚665 .loss/FRt4/detection_loss‚2.316352801431309 epoch‚6 % _timestamp‚1.7884443809221592e+09 0loss/FRt1/assignment_loss‚0.7154236011780225 /loss/FRt3/detection_loss‚2.5271483439737494 0loss/SRqqt3/detection_loss‚2.097048160997597)»#»]Î × &loss/total_loss‚39.271448182049056 /loss/FRt2/assignment_loss‚2.549296262652375 2loss/SRqqt2/assignment_loss‚1.3871129742315547 2loss/FBt2/assignment_loss‚0.014897083743282845 .loss/FRt2/detection_loss‚2.333020806312561 -loss/FBt3/detection_loss‚2.59521244764328 0loss/FRt4/assignment_loss‚3.0100572716106067 0loss/FBt4/assignment_loss‚0.6840788976331169 .loss/FRt1/detection_loss‚1.313333108376425 .loss/FBt1/detection_loss‚0.520511554215442 _step‚665 .loss/FRt4/detection_loss‚2.316352801431309 epoch‚6 % _timestamp‚1.7884443809221592e+09 0loss/FRt1/assignment_loss‚0.7154236011780225 /loss/FRt3/detection_loss‚2.5271483439737494 0loss/SRqqt3/detection_loss‚2.097048160997597 1loss/FBt3/assignment_loss‚0.18095530279388186 /loss/FBt4/detection_loss‚2.3926612794399262 trainer/global_step‚8099 /loss/FRt3/assignment_loss‚3.348271315296491 1loss/SRqqt1/assignment_loss‚0.785258673988874 2loss/SRqqt3/assignment_loss‚1.5958505370024894 1loss/SRqqt4/detection_loss‚1.7065003492214061 .loss/FBt2/detection_loss‚2.799139371285072 2loss/SRqqt4/assignment_loss‚1.5825270535079417 0loss/FBt1/assignment_loss‚0.1797649154058476 1loss/SRqqt1/detection_loss‚0.7714629253697773 1loss/SRqqt2/detection_loss‚1.8655631447380239 _runtime‚6820.966740711™(è3”Ï ‚(:‰ àýåÔ轉É gpu.0.gpu‚78.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚278.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦èÊvDÐ ‚(:¹ àýåÔÓ‡€Ë disk.drbd1777.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚566.45751953125 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.310233120697408 network.recv‚ 2.470642e+06 disk.dm-1.out‚ 249.83203125# disk.nvme0c0n1.in‚ 1754.82421875# disk.dm-0.out‚58833.73388671875 disk.drbd1777.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚58833.73388671875 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.373046875 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 1777.671875( disk./.usagePercent‚74.3718448095435 disk.dm-0.in‚ 1754.82421875! disk.nvme0n1.in‚ 1754.82421875& disk.nvme0n1.out‚58833.73388671875 disk.rbd0.in‚0$ memory_percent‚89.14995193481445 disk.rbd1.in‚0) proc.memory.percent‚16.06571674346924 disk.sda.out‚ 249.83203125 disk.sda3.out‚ 249.83203125# proc.memory.rssMB‚ 2632.20703125 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.rbd0.out‚0 network.sent‚ 2.0109596e+07% disk./.usageGB‚1108.1942329406738 disk.sda2.in‚0@¿ ®“Ñ ‚(:ˆ ïýåÔÍÝ›É gpu.0.gpu‚71.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚98.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾U7CÒ ‚(:¸ ïýåÔÙœïÊ disk.rbd2.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1754.82421875 disk.dm-1.in‚0.5625$ memory_percent‚89.14930820465088 disk.sda1.in‚0! disk.nvme0n1.in‚ 1754.82421875& disk.nvme0n1.out‚58834.40673828125 disk.sda3.in‚0.5625 disk.dm-1.out‚ 250.5078125) proc.memory.percent‚16.06571674346924 network.recv‚ 2.477511e+06 disk.rbd1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2632.20703125 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0 cpu‚25.30962930461403 disk.drbd1777.in‚0 disk.sda2.in‚0# disk.dm-0.out‚58834.40673828125) proc.memory.availableMB‚ 1777.77734375! disk.rbd2.out‚566.45751953125 disk.sda.in‚0.5625 disk.dm-0.in‚ 1754.82421875 network.sent‚ 2.0219363e+07% disk./.usageGB‚1108.1942710876465 disk.sda.out‚ 250.5078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 250.5078125( disk.nvme0c0n1.out‚58834.40673828125) disk./.usagePercent‚74.37184736961885_‘èÐÓ ‚(:’ þýåÔÝ€É gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.64' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.12799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§39Ô ‚(:® þýåÔ·éÚÊ' disk.nvme0c0n1.out‚58835.0185546875 disk.rbd1.in‚0' proc.memory.availableMB‚ 1777.765625) proc.memory.percent‚16.06571674346924 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚568.19970703125# proc.memory.rssMB‚ 2632.20703125 disk.dm-0.in‚ 1755.15625 disk.dm-1.in‚0.5625 disk.sda1.in‚0) disk./.usagePercent‚74.37187220234966 disk.nvme0n1.in‚ 1755.15625% disk.nvme0n1.out‚58835.0185546875 disk.sda.in‚0.5625 disk.drbd1777.out‚0$ memory_percent‚89.14937973022461 cpu‚25.309064601001282 disk.nvme0c0n1.in‚ 1755.15625 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 250.78515625 network.sent‚ 2.0224584e+07 disk.sda3.out‚ 250.78515625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0" disk.dm-0.out‚58835.0185546875 disk.dm-1.out‚ 250.78515625 disk.drbd1777.in‚0 disk.sda1.out‚0 network.recv‚ 2.481246e+06% disk./.usageGB‚1108.1946411132812 disk.rbd3.in‚0裥ß2Õ - _wandbruntime‚6859  _runtime‚6859â/SÉÖ Ã trainer/global_step‚8149 % _timestamp‚1.7884444190776484e+09 _runtime‚6859.122349621 _step‚666 ' lr-AdamW/pg1‚0.00011246798434926843 ' lr-AdamW/pg2‚0.00011246798434926843s*jÎ× È _runtime‚6859.122349621 _step‚666 ' lr-AdamW/pg1‚0.00011246798434926843 ' lr-AdamW/pg2‚0.00011246798434926843 trainer/global_step‚8149 % _timestamp‚1.7884444190776484e+09šjWé¡2Ø - _wandbruntime‚6859  _runtime‚6859¶Ý‹¤(Ù ŠP XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15113647498240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚922741151865856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884444190784261e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272035332 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚797153527885312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _runtime‚6859.123315251 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77947286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893174645349376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚922738525264896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151886449 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚907624904158208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20706471060992 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134388907 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚907624877766656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚797153449160192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15091201500652 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 trainer/global_step‚8149 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚922741151865856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚922738525264896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚907624877766656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120148883 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77947283 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20706521651712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15088603215984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15116247707648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151906483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15116247707648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56441621 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚908263248565360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚907624904158208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20706471060992 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134388907 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚907624877766656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚797153449160192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15091201500652 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 trainer/global_step‚8149 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚922741151865856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚922738525264896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚907624877766656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120148883 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77947283 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20706521651712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15088603215984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15116247707648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151906483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15116247707648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56441621 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚908263248565360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120163119 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151924743 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77956479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚797248346765312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚893281011951616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776539366152704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚907732817472000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15118080049664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151904683 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77956482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 % _timestamp‚1.7884444198130178e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067802 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120163119 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20709031190016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚922850923913216 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15093030843528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚907732843863552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚797248425477120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20708980612608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 trainer/global_step‚8149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚907732817472000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56448354 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120163125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134404361 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚908374042795144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚907732817472000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15118080049664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151904683 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77956482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 % _timestamp‚1.7884444198130178e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067802 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120163119 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20709031190016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚922850923913216 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15093030843528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚907732843863552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚797248425477120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20708980612608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 trainer/global_step‚8149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚907732817472000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56448354 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120163125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134404361 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚908374042795144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0œ‚¹Û42Þ - _wandbruntime‚6859  _runtime‚6859ð+ý[ß Õ 2loss/SRqqt1/assignment_loss‚0.7607622831729391 1loss/SRqqt3/assignment_loss‚1.269167245749165 1loss/FBt1/assignment_loss‚0.09487901211303101 /loss/FRt1/detection_loss‚1.7088373411624205 /loss/FRt3/detection_loss‚2.2940921739052125 .loss/FBt4/detection_loss‚2.365992854941975 &loss/total_loss‚38.577717944492804 epoch‚6 1loss/SRqqt4/assignment_loss‚1.329767931318074 /loss/FRt2/detection_loss‚2.5364279563610372 1loss/SRqqt1/detection_loss‚1.0042168738380555 /loss/FBt2/detection_loss‚2.0131913790336022 trainer/global_step‚8149 _step‚669 2loss/SRqqt2/assignment_loss‚1.4570253038697916 0loss/FRt1/assignment_loss‚1.4181173685051145 /loss/FRt3/assignment_loss‚3.108976467235668 1loss/SRqqt2/detection_loss‚1.8062885627150536 0loss/FRt2/assignment_loss‚2.8666372069945703 1loss/FBt3/assignment_loss‚0.25156126680990887 1loss/SRqqt4/detection_loss‚1.6098976816263115 0loss/FRt4/assignment_loss‚2.4743735760450365 /loss/FBt1/detection_loss‚0.4007336654123554 0loss/FBt4/assignment_loss‚0.5117923431310125 _runtime‚ 6859.85936757 % _timestamp‚1.7884444198147895e+09 1loss/FBt2/assignment_loss‚0.45603374294184434 -loss/FRt4/detection_loss‚2.49665679782629 0loss/SRqqt3/detection_loss‚2.140250654781566 /loss/FBt3/detection_loss‚2.2020382550027637‘Ñûí`à Ú 0loss/FRt4/assignment_loss‚2.4743735760450365 /loss/FBt1/detection_loss‚0.4007336654123554 0loss/FBt4/assignment_loss‚0.5117923431310125 _runtime‚ 6859.85936757 % _timestamp‚1.7884444198147895e+09 1loss/FBt2/assignment_loss‚0.45603374294184434 -loss/FRt4/detection_loss‚2.49665679782629 0loss/SRqqt3/detection_loss‚2.140250654781566 /loss/FBt3/detection_loss‚2.2020382550027637 2loss/SRqqt1/assignment_loss‚0.7607622831729391 1loss/SRqqt3/assignment_loss‚1.269167245749165 1loss/FBt1/assignment_loss‚0.09487901211303101 /loss/FRt1/detection_loss‚1.7088373411624205 /loss/FRt3/detection_loss‚2.2940921739052125 .loss/FBt4/detection_loss‚2.365992854941975 &loss/total_loss‚38.577717944492804 epoch‚6 1loss/SRqqt4/assignment_loss‚1.329767931318074 /loss/FRt2/detection_loss‚2.5364279563610372 1loss/SRqqt1/detection_loss‚1.0042168738380555 /loss/FBt2/detection_loss‚2.0131913790336022 trainer/global_step‚8149 _step‚669 2loss/SRqqt2/assignment_loss‚1.4570253038697916 0loss/FRt1/assignment_loss‚1.4181173685051145 /loss/FRt3/assignment_loss‚3.108976467235668 1loss/SRqqt2/detection_loss‚1.8062885627150536 0loss/FRt2/assignment_loss‚2.8666372069945703 1loss/FBt3/assignment_loss‚0.25156126680990887 1loss/SRqqt4/detection_loss‚1.6098976816263115÷3€”á ‚(:‰ þåÔ°Ì†É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚158.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕKÛ•5â ‚(:ª þåÔþç­Ë disk.sda.in‚0.5625 disk.sda3.in‚0.5625' disk.nvme0c0n1.out‚58933.2568359375 disk.dm-0.in‚ 1755.15625# proc.memory.rssMB‚ 2632.20703125% disk./.usageGB‚1108.1937522888184 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0) disk./.usagePercent‚74.37181255259418 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 cpu‚25.308496253936024 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0# proc.memory.availableMB‚1777.75 network.sent‚ 2.0230745e+07 disk.sda.out‚ 251.44140625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1755.15625 disk.nvme0c0n1.in‚ 1755.15625 disk.dm-1.in‚0.5625$ memory_percent‚89.14947509765625 network.recv‚ 2.485648e+06 disk.sda2.out‚0% disk.nvme0n1.out‚58933.2568359375" disk.dm-0.out‚58933.2568359375 disk.dm-1.out‚ 251.44140625) proc.memory.percent‚16.06571674346924! disk.rbd2.out‚568.26220703125 disk.sda3.out‚ 251.44140625 disk.drbd1777.in‚0 disk.rbd0.out‚0¤Ñ#Ÿã ‚(:” œþåÔÑ¿–É gpu.0.gpu‚58.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚222.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.444799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“¾Ã‰=ä ‚(:² œþåÔ–ÁäÌ network.sent‚ 2.0340219e+07 disk.drbd1777.in‚0 disk.sda3.out‚ 251.80078125 network.recv‚ 2.492674e+06 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875 disk.rbd2.in‚0! disk.rbd2.out‚569.93798828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 1755.15625 cpu‚25.30792442877873 disk.rbd1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚74.37180896848871% disk./.usageGB‚1108.1936988830566 disk.sda.in‚0.5625 disk.rbd0.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚58933.60498046875 disk.sda.out‚ 251.80078125& disk.nvme0n1.out‚58933.60498046875 disk.dm-1.out‚ 251.80078125( disk.nvme0c0n1.out‚58933.60498046875 disk.dm-0.in‚ 1755.15625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1755.15625 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 1777.75390625$ memory_percent‚89.14945125579834# proc.memory.rssMB‚ 2632.20703125) proc.memory.percent‚16.06571674346924?ˆÊ$2å - _wandbruntime‚6898  _runtime‚6898×=˜«Éæ à ' lr-AdamW/pg1‚0.00010993751020430902 ' lr-AdamW/pg2‚0.00010993751020430902 trainer/global_step‚8199 % _timestamp‚1.7884444582159052e+09 _runtime‚6898.260644613 _step‚670ždPˆÎç È ' lr-AdamW/pg2‚0.00010993751020430902 trainer/global_step‚8199 % _timestamp‚1.7884444582159052e+09 _runtime‚6898.260644613 _step‚670 ' lr-AdamW/pg1‚0.00010993751020430902ž½|†Z2è - _wandbruntime‚6898  _runtime‚6898Ó§"ÎÃé ‰P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚801897993605632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚928229767982080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15207878556672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚928227141381120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135184682 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15180083940592 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152801501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚928227141381120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚928229767982080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20831536203264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273682224 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6898.261537516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78407083 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120860689 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152821535 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78407086 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚898491704062976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚913021863033856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚781066408908800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56777596 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898491729490944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15205278347264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 $ _timestamp‚1.788444458216641e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_b»TjžLytes.large_pool.allocated‚913021889425408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120860683 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15207878556672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚913671788003568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152801501 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20831584696832 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273662184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚801897916977664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15182682225260 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚913021863033856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120860689 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649&WJõ(ê ŽP ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚801897916977664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15182682225260 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚913021863033856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120860689 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚801897993605632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚928229767982080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15207878556672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚928227141381120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135184682 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15180083940592 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152801501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚928227141381120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚928229767982080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20831536203264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273682224 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6898.261537516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78407083 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120860689 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152821535 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78407086 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2701149184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚898491704062976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚913021863033856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚781066408908800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56777596 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898491729490944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15205278347264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 $ _timestamp‚1.788444458216641e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚913021889425408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120860683 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15207878556672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚913671788003568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152801501 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20831584696832 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273662184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034Ÿ2ë - _wandbruntime‚6899  _runtime‚6899òôS•'ì ”N ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚898598070665216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273694654 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56784236 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚781158796962304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135200072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 trainer/global_step‚8199 % _timestamp‚1.7884444589590578e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15209710898688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152819735 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚913779958520508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚801992897620480 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚781158768827904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152819735 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120874919 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚928336913415168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20834052177920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚928339540029440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694654 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78416282 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120874919 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15181913283260 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚913129802739200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913129802739200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56783793 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120874925 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚913129829130752 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚913782582233352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898598045237248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273694654 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78416282 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120874919 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15181913283260 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚913129802739200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913129802739200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56783793 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120874925 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚913129829130752 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚913782582233352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898598045237248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135200072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 trainer/global_step‚8199 % _timestamp‚1.7884444589590578e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15209710898688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152819735 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚913779958520508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚801992897620480 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚781158768827904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152819735 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2701149184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120874919 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.cur%ßÀîrent‚ 2600222720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚928336913415168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20834052177920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚928339540029440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15274162949868 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15296909196288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153716553 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚785685811584000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15296909196288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135980504 % _timestamp‚1.7884444972691119e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933718384098304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933715757497344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78866886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚485 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919080327441776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚903808762776576 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933715757497344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚918418874692608 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20957130118656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135980016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚785685783449600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15271564665200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121572483 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933718384098304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15299509405696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57113133 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153736587 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57113618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275289036 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15299509405696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121572483 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚918418874692608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933715757497344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚918418874692608 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20957130118656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135980016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚785685783449600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15271564665200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121572483 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933718384098304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15299509405696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57113133 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153736587 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57113618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275289036 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15299509405696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121572483 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚918418874692608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15274162949868 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6£*ס02ý - _wandbruntime‚6938  _runtime‚6938O÷ƒÎþ “N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57120362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121586719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15301341747712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚785778199637504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15298741524992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135996444 $ _timestamp‚1.788444498008682e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20959592657408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚918526814397952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚918526788006400 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135995983 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121586719 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚6938.053876516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153734787 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15301341747712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚785778171503104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153754847 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚918526814397952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57119904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78876079 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚806737840775168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚806737764160512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121586725 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933828156145664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78876082 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919188497958716 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15275992292744 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚676 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153754847 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20959641137664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚903915129378816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121586725 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚919191121671560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933828156145664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275321506 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933825529531392 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚918526788006400 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 trainer/global_step‚8249Ÿ5#¤'ÿ ˜N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚6938.053876516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153734787 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15301341747712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚785778171503104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153754847 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚918526814397952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57119904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78876079 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚806737840775168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚806737764160512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121586725 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933828156145664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78876082 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919188497958716 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15275992292744 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚676 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153754847 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20959641137664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚903915129378816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121586725 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚919191121671560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933828156145664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 382103552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275321506 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933825529531392 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚918526788006400 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 trainer/global_step‚8249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57120362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121586719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15301341747712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚785778199637504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15298741524992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135996444 $ _timestamp‚1.788444498008682e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20959592657408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚918526814397952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚918526788006400 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135995983 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121586719 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506¤…Ö2€!- _wandbruntime‚6938  _runtime‚6938Š1H¸Z!Ô 0loss/FRt1/assignment_loss‚1.1092524548708382 0loss/FRt3/assignment_loss‚3.6299442992065893 2loss/SRqqt1/assignment_loss‚0.8657706821955239 2loss/SRqqt2/assignment_loss‚1.4586131255693582 1loss/FBt3/assignment_loss‚0.21158272569300607 0loss/SRqqt1/detection_loss‚0.926802346110344 /loss/FBt1/detection_loss‚0.4236103818416162 trainer/global_step‚8249 2loss/SRqqt3/assignment_loss‚1.5914045661207168 epoch‚6 _step‚677 .loss/FBt2/detection_loss‚2.612197526863643 %loss/total_loss‚40.66479845878619 0loss/FRt2/assignment_loss‚2.5406030208565467 1loss/FBt1/assignment_loss‚0.15133893186412806 .loss/FRt2/detection_loss‚2.525689261250718 /loss/FBt4/detection_loss‚2.2734028875827788 1loss/FBt2/assignment_loss‚0.26570935789121514 1loss/SRqqt3/detection_loss‚2.3931043051904246 1loss/SRqqt2/detection_loss‚1.9091689203466689 _runtime‚6938.055021895 /loss/FRt4/assignment_loss‚2.542360241315803 1loss/SRqqt4/assignment_loss‚1.457160727963561 .loss/FRt3/detection_loss‚2.821636194916596 /loss/FRt4/detection_loss‚2.0144688207276014 % _timestamp‚1.7884444980104446e+09 1loss/FBt4/assignment_loss‚0.40006441860459746 -loss/FRt1/detection_loss‚1.55360525808597 0loss/SRqqt4/detection_loss‚2.068236955219791 /loss/FBt3/detection_loss‚2.9190710484981537{…Ñ&_‚!Ù 1loss/FBt4/assignment_loss‚0.40006441860459746 -loss/FRt1/detection_loss‚1.55360525808597 0loss/SRqqt4/detection_loss‚2.068236955219791 /loss/FBt3/detection_loss‚2.9190710484981537 0loss/FRt1/assignment_loss‚1.1092524548708382 0loss/FRt3/assignment_loss‚3.6299442992065893 2loss/SRqqt1/assignment_loss‚0.8657706821955239 2loss/SRqqt2/assignment_loss‚1.4586131255693582 1loss/FBt3/assignment_loss‚0.21158272569300607 0loss/SRqqt1/detection_loss‚0.926802346110344 /loss/FBt1/detection_loss‚0.4236103818416162 trainer/global_step‚8249 2loss/SRqqt3/assignment_loss‚1.5914045661207168 epoch‚6 _step‚677 .loss/FBt2/detection_loss‚2.612197526863643 %loss/total_loss‚40.66479845878619 0loss/FRt2/assignment_loss‚2.5406030208565467 1loss/FBt1/assignment_loss‚0.15133893186412806 .loss/FRt2/detection_loss‚2.525689261250718 /loss/FBt4/detection_loss‚2.2734028875827788 1loss/FBt2/assignment_loss‚0.26570935789121514 1loss/SRqqt3/detection_loss‚2.3931043051904246 1loss/SRqqt2/detection_loss‚1.9091689203466689 _runtime‚6938.055021895 /loss/FRt4/assignment_loss‚2.542360241315803 1loss/SRqqt4/assignment_loss‚1.457160727963561 .loss/FRt3/detection_loss‚2.821636194916596 /loss/FRt4/detection_loss‚2.0144688207276014 % _timestamp‚1.7884444980104446e+09¥= q“ƒ!‚(:ˆ ØþåÔ›‚ˆÉ gpu.0.gpu‚70.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚256.475' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë•G@„!‚(:µ ØþåÔ묶Ì disk.rbd0.out‚0 disk.dm-0.in‚ 1755.46484375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 253.6171875 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 253.6171875) proc.memory.percent‚16.06571674346924) disk./.usagePercent‚74.30234772426097 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1777.796875# proc.memory.rssMB‚ 2632.20703125$ disk./.usageGB‚1107.158676147461 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1755.46484375( disk.nvme0c0n1.out‚59196.17919921875 proc.cpu.threads‚17 network.recv‚ 2.512301e+06! disk.rbd2.out‚573.41845703125 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1755.46484375& disk.nvme0n1.out‚59196.17919921875 network.sent‚ 2.0466943e+07 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.30567158490886$ memory_percent‚89.14918899536133# disk.dm-0.out‚59196.17919921875 disk.dm-1.out‚ 253.6171875¸1О…!‚(:“ çþåÔщüÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚240.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.114000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥/ŸÑ;†!‚(:° çþåÔöÉ‹Ë# disk.nvme0c0n1.in‚ 1755.46484375# disk.dm-0.out‚59196.70361328125 disk.drbd1777.in‚0 network.sent‚ 2.0576661e+07 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.out‚0! disk.rbd2.out‚574.99267578125) proc.memory.availableMB‚ 1777.54296875( disk.nvme0c0n1.out‚59196.70361328125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda.in‚0.5625 disk.sda.out‚ 254.09375 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.percent‚16.06571674346924) disk./.usagePercent‚74.30228039427945 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 254.09375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2632.20703125 cpu‚25.30513267087884 disk.sda3.out‚ 254.09375 disk.rbd3.out‚0& disk.nvme0n1.out‚59196.70361328125$ memory_percent‚89.15073871612549# disk./.usageGB‚1107.15767288208 disk.rbd0.in‚0 network.recv‚ 2.519215e+06! disk.nvme0n1.in‚ 1755.46484375 disk.rbd2.in‚0™ã攇!‚(:‰ öþåÔþ“É gpu.0.gpu‚53.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚249.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.9956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¶œfCˆ!‚(:¸ öþåÔµÿ—Ì) proc.memory.availableMB‚ 1777.41015625 disk.sda.out‚ 254.55859375 disk.sda2.out‚0 disk.sda3.out‚ 254.55859375# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375 disk.dm-1.out‚ 254.55859375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0" disk.dm-0.out‚59197.0732421875$ memory_percent‚89.15154933929443 proc.cpu.threads‚17 network.recv‚ 2.522756e+06! disk.rbd2.out‚576.61767578125 disk.sda2.in‚0# proc.memory.rssMB‚ 2632.20703125) proc.memory.percent‚16.06571674346924) disk./.usagePercent‚74.30225888964658% disk./.usageGB‚1107.1573524475098 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.sda1.in‚0' disk.nvme0c0n1.out‚59197.0732421875 network.sent‚ 2.0581878e+07% disk.nvme0n1.out‚59197.0732421875 disk.rbd0.out‚0 cpu‚25.30461352233993 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1755.46484375W3ë¨2‰!- _wandbruntime‚6976  _runtime‚6976¯Z?,ÉŠ!à _step‚678 ' lr-AdamW/pg1‚0.00010492948921865834 ' lr-AdamW/pg2‚0.00010492948921865834 trainer/global_step‚8299 % _timestamp‚1.7884445364748018e+09 _runtime‚6976.519514115ëòSä΋!È ' lr-AdamW/pg2‚0.00010492948921865834 trainer/global_step‚8299 % _timestamp‚1.7884445364748018e+09 _runtime‚6976.519514115 _step‚678 ' lr-AdamW/pg1‚0.00010492948921865834¦˪@w2Œ!- _wandbruntime‚6976  _runtime‚6976áB¯¾!ŠP \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚923815833568256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚909125821490176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122284283 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276935928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154631605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21081724055040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15363045389808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚939204373613568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15388540045312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚924488866879984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚923815859959808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79326686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15391140254720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21081671367168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚939207000214528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚923815859959808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154651639 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15388540045312 trainer/global_step‚8299 % _timestamp‚1.7884445364756207e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚939207000214528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136778429 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15391140254720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚790305186124800 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚6976.520449748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915888 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚909125846918144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122284289 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚923815833568256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122284283 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 BÛDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚939204373613568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57451743 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚924491490592620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79326683 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57451221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚790305214259200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136777904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚811386938314240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚811386857491968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚679 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276915888 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚909125846918144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122284289 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚923815833568256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122284283 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚939204373613568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57451743 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚924491490592620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79326683 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57451221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚790305214259200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136777904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚811386938314240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚811386857491968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚679 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚924488866879984 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚923815859959808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79326686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15391140254720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21081671367168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚939207000214528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚923815859959808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154651639 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15388540045312 trainer/global_step‚8299 % _timestamp‚1.7884445364756207e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚939207000214528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 382103552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136778429 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074§oÖ¹/2!- _wandbruntime‚6977  _runtime‚6977(šÌã'!“N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136794208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57457830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21084220331008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚790397602312704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122298525 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15392972596736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚924597037396924 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154649839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79335879 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15392972596736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909232188092416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚909232162664448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154669899 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21084167656448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚939316772261888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15364874732476 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122298525 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚6977.274781617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136793709 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15390372374016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚939316772261888 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚811481822643712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154649839 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 $ _timestamp‚1.788444537229576e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122298519 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811481741834752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575/ɃR+#‘!˜N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚939316772261888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15364874732476 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122298525 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚6977.274781617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136793709 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15390372374016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚939316772261888 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚811481822643712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154649839 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 $ _timestamp‚1.788444537229576e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122298519 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811481741834752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136794208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57457830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21084220331008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚790397602312704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122298525 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15392972596736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚924597037396924 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154649839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79335879 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15392972596736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909232188092416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚909232162664448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.y+Ž>ósmall_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154669899 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21084167656448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¨þJÉ2’!- _wandbruntime‚6977  _runtime‚6977›<„.V“!Ð 1loss/SRqqt1/detection_loss‚0.8985276961466298 /loss/FBt1/detection_loss‚0.4409464598367257 .loss/FBt4/detection_loss‚5.076685473322868 #loss/total_loss‚40.949338386576 epoch‚6 /loss/FRt2/detection_loss‚2.3990630817179586 0loss/FBt1/assignment_loss‚0.1188376459540758 1loss/FBt2/assignment_loss‚0.15305919936508872 2loss/FBt3/assignment_loss‚0.015267067661625333 .loss/FRt4/detection_loss‚2.474502815410148 0loss/FBt4/assignment_loss‚0.2203819517071679 1loss/SRqqt4/detection_loss‚1.8220990414917468 trainer/global_step‚8299 _step‚681 0loss/FRt2/assignment_loss‚3.2033494827794096 /loss/FRt4/assignment_loss‚2.438773200955502 .loss/FRt1/detection_loss‚1.434506184614773 $ _timestamp‚1.788444537231335e+09 2loss/SRqqt1/assignment_loss‚0.8364945216453634 1loss/SRqqt4/assignment_loss‚1.163389016762376 /loss/FBt2/detection_loss‚2.0760077089071274 /loss/FRt1/assignment_loss‚1.170693650560559 .loss/FRt3/detection_loss‚2.446726674282485 _runtime‚6977.275895276 2loss/SRqqt2/assignment_loss‚1.2478769192428782 1loss/SRqqt3/assignment_loss‚1.456429804765409 /loss/FBt3/detection_loss‚2.5602127611637115 /loss/FRt3/assignment_loss‚2.889385100673227 1loss/SRqqt2/detection_loss‚2.0202367016395857 1loss/SRqqt3/detection_loss‚2.3858862259695606и.Ò[”!Õ 1loss/SRqqt2/detection_loss‚2.0202367016395857 1loss/SRqqt3/detection_loss‚2.3858862259695606 1loss/SRqqt1/detection_loss‚0.8985276961466298 /loss/FBt1/detection_loss‚0.4409464598367257 .loss/FBt4/detection_loss‚5.076685473322868 #loss/total_loss‚40.949338386576 epoch‚6 /loss/FRt2/detection_loss‚2.3990630817179586 0loss/FBt1/assignment_loss‚0.1188376459540758 1loss/FBt2/assignment_loss‚0.15305919936508872 2loss/FBt3/assignment_loss‚0.015267067661625333 .loss/FRt4/detection_loss‚2.474502815410148 0loss/FBt4/assignment_loss‚0.2203819517071679 1loss/SRqqt4/detection_loss‚1.8220990414917468 trainer/global_step‚8299 _step‚681 0loss/FRt2/assignment_loss‚3.2033494827794096 /loss/FRt4/assignment_loss‚2.438773200955502 .loss/FRt1/detection_loss‚1.434506184614773 $ _timestamp‚1.788444537231335e+09 2loss/SRqqt1/assignment_loss‚0.8364945216453634 1loss/SRqqt4/assignment_loss‚1.163389016762376 /loss/FBt2/detection_loss‚2.0760077089071274 /loss/FRt1/assignment_loss‚1.170693650560559 .loss/FRt3/detection_loss‚2.446726674282485 _runtime‚6977.275895276 2loss/SRqqt2/assignment_loss‚1.2478769192428782 1loss/SRqqt3/assignment_loss‚1.456429804765409 /loss/FBt3/detection_loss‚2.5602127611637115 /loss/FRt3/assignment_loss‚2.889385100673227©È‹=“•!‚(:ˆ …ÿåÔ‰°žÉ gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚230.845' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.169 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯yß=–!‚(:² …ÿåÔ™«•Ë network.sent‚ 2.0587923e+07 disk.sda2.in‚0 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.526944e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46484375 disk.dm-1.out‚ 255.0234375# proc.memory.rssMB‚ 2632.20703125) proc.memory.percent‚16.06571674346924! disk.rbd2.out‚576.61767578125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1755.46484375) proc.memory.availableMB‚ 1777.48046875 disk.sda1.out‚0& disk.nvme0c0n1.out‚59197.947265625) disk./.usagePercent‚74.30229985085207% disk./.usageGB‚1107.1579627990723 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 255.0234375! disk.nvme0n1.in‚ 1755.46484375 disk.sda3.in‚0.5625$ memory_percent‚89.15112018585205 disk.drbd1777.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚59197.947265625 disk.sda3.out‚ 255.0234375! disk.dm-0.out‚59197.947265625 cpu‚25.30404130109679Y'vã’—!‚(:‡ ”ÿåÔø¿ÝÈ gpu.0.gpu‚72.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚186.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ür=˜!‚(:² ”ÿåÔ´½®Ì disk.sda3.out‚ 255.3359375& disk.nvme0c0n1.out‚59198.291015625 disk.dm-1.in‚0.5625 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚59198.291015625 disk.sda.in‚0.5625# proc.memory.rssMB‚ 2632.20703125 network.sent‚ 2.0696428e+07) disk./.usagePercent‚74.30229626674658! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.out‚0 disk.rbd1.in‚0! disk.rbd2.out‚578.30908203125$ memory_percent‚89.15057182312012( proc.memory.availableMB‚ 1777.5703125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 cpu‚25.303490524133924% disk./.usageGB‚1107.1579093933105 disk.sda3.in‚0.5625! disk.dm-0.out‚59198.291015625 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 255.3359375 disk.rbd3.in‚0 network.recv‚ 2.533172e+06# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375 disk.dm-1.out‚ 255.3359375 disk.sda1.in‚0) proc.memory.percent‚16.06571674346924”¹F2™!- _wandbruntime‚7016  _runtime‚7016¨š‚Åš!¿ ' lr-AdamW/pg1‚0.00010245298604203416 ' lr-AdamW/pg2‚0.00010245298604203416 trainer/global_step‚8349 $ _timestamp‚1.788444576476526e+09 _runtime‚ 7016.521173 _step‚682³kzÊ›!Ä $ _timestamp‚1.788444576476526e+09 _runtime‚ 7016.521173 _step‚682 ' lr-AdamW/pg1‚0.00010245298604203416 ' lr-AdamW/pg2‚0.00010245298604203416 trainer/global_step‚8349ª íÍ2œ!- _wandbruntime‚7016  _runtime‚7016™©ŽØ(!ŠP M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137573714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚816130606065152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 % _timestamp‚1.7884445764772553e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚794924616934400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15482771103744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8349 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚929900030030828 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚944692989729792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚929212818835456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15480170894336 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15457124399084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚944695616330752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15480170894336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122996083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57787231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21205936442880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155546657 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79786486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚929212818835456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚929212845227008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚929897406318192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122996089 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79786483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137574207 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776Hæhø(ž!P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15480170894336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122996083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57787231 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21205936442880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155546657 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79786486 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚929212818835456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚929212845227008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚929897406318192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122996089 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79786483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137574207 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137573714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚816130606065152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 % _timestamp‚1.7884445764772553e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚794924616934400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15482771103744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8349 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚929900030030828 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚944692989729792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚929212818835456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15480170894336 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15457124399084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚944695616330752«¬\•2Ÿ!- _wandbruntime‚7017  _runtime‚7017OpEüÑ !”N S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930005576835132 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚795017004987904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15482003223040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123010319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚930008200547976 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15456355457084 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137589672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21208432467456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123010325 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚929320758540800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123010325 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21208485142016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155564891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79795682 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595276 B8DeviceStatsMonitor.on_train_batch_ï-©Iend/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚944805388378112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚816225490129920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15458953741960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15482003223040 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 trainer/global_step‚8349 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155564891 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57793993 % _timestamp‚1.7884445774719336e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚929320784932352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15484603445760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚816225409320960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 _step‚684 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚944802761763840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚914549246806016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155584951 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚929320758540800 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15456355457084 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137589672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21208432467456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123010325 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚929320758540800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123010325 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21208485142016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155564891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79795682 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚944805388378112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚816225490129920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15458953741960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15482003223040 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 trainer/global_step‚8349 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155564891 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57793993 % _timestamp‚1.7884445774719336e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚929320784932352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15484603445760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚816225409320960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 _step‚684 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚944802761763840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚914549246806016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155584951 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚929320758540800 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934609804102656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138370517 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚950181605846016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820875304748032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934609830494208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 % _timestamp‚1.7884446166088524e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80246283 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123707889 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156481743 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799544019609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15574401952768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58124234 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _runtime‚ 7056.6538415 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚950184232446976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚919759938917376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚950181605846016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934609804102656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123707889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156481743 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156461709 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚919759964345344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚950184232446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80246286 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820875385570304 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123707883 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15574401952768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚799543991475200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15546006839024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123707883 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156461709 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280189632 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15571801743360 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚935308569469036 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21331365960704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚935305945756400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15548605123692 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58124760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21331313272832 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStat¿¯XË_sMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 _step‚687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934609830494208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776Ïi(°!P GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123707883 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156461709 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280189632 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15571801743360 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚935308569469036 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21331365960704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚935305945756400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15548605123692 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58124760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21331313272832 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 _step‚687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934609830494208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 trainer/global_step‚8399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169592 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138371046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934609804102656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138370517 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚950181605846016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚820875304748032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934609830494208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 % _timestamp‚1.7884446166088524e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80246283 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123707889 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156481743 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799544019609600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15574401952768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58124234 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _runtime‚ 7056.6538415 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚950184232446976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚919759938917376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚950181605846016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934609804102656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123707889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156481743 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156461709 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚919759964345344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚950184232446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80246286 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820875385570304 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123707883 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15574401952768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚799543991475200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15546006839024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408¯^ Z=2±!- _wandbruntime‚7057  _runtime‚7057qѳV'²!”N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123722125 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123722119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15573634072064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚935416739986184 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156479943 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚919866305519616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138386821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934717770199552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚950291377880064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934717770199552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7057.400262242 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156500003 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚688 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156479943 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156500003 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚799636379528704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123722125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138386307 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934717743808000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80255482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799636407663104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚950294004494336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80255482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799636407663104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚950294004494336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚935416739986184 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156479943 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚919866305519616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138386821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934717770199552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚950291377880064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934717770199552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7057.400262242 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156500003 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚688 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156479943 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156500003 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚799636379528704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚54771712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123722125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138386307 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934717743808000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575°Â9ñâ2´!- _wandbruntime‚7057  _runtime‚7057•ÊhÒ`µ!Ú 0loss/FRt4/assignment_loss‚2.5056781946001827 epoch‚6 /loss/FRt2/detection_loss‚2.3214318556918037 /loss/FRt3/assignment_loss‚3.565585380015166 0loss/SRqqt4/detection_loss‚1.599685728008097 .loss/FBt3/detection_loss‚4.072693854570389 0loss/FRt1/assignment_loss‚0.9510548757879358 .loss/FRt2/assignment_loss‚3.08449686633216 2loss/SRqqt3/assignment_loss‚1.5788639264646918 1loss/FBt1/assignment_loss‚0.14091884312612799 3loss/FBt3/assignment_loss‚0.0063319285691250116 1loss/SRqqt1/detection_loss‚1.1037162878801898 /loss/FBt2/detection_loss‚2.3648262725156894 /loss/FBt4/detection_loss‚1.8902790082825556 2loss/SRqqt2/assignment_loss‚1.2745790967234858 0loss/FBt4/assignment_loss‚0.2540609834217725 /loss/FRt4/detection_loss‚2.6165892248218126 /loss/FBt1/detection_loss‚0.5841165998211021 % _timestamp‚1.7884446173571126e+09 _step‚689 /loss/FRt1/detection_loss‚1.9357547252193879 2loss/SRqqt1/assignment_loss‚1.0122584424735535 2loss/SRqqt4/assignment_loss‚1.6832345672629097 1loss/SRqqt2/detection_loss‚1.6635954109606919 _runtime‚7057.401685664 1loss/FBt2/assignment_loss‚0.13670118807257334 1loss/SRqqt3/detection_loss‚2.6315561309456825 trainer/global_step‚8399 /loss/FRt3/detection_loss‚3.0081818840403387 &loss/total_loss‚41.986191275607425PГ Y¶!ß 1loss/FBt2/assignment_loss‚0.13670118807257334 1loss/SRqqt3/detection_loss‚2.6315561309456825 trainer/global_step‚8399 /loss/FRt3/detection_loss‚3.0081818840403387 &loss/total_loss‚41.986191275607425 0loss/FRt4/assignment_loss‚2.5056781946001827 epoch‚6 /loss/FRt2/detection_loss‚2.3214318556918037 /lo];/X ss/FRt3/assignment_loss‚3.565585380015166 0loss/SRqqt4/detection_loss‚1.599685728008097 .loss/FBt3/detection_loss‚4.072693854570389 0loss/FRt1/assignment_loss‚0.9510548757879358 .loss/FRt2/assignment_loss‚3.08449686633216 2loss/SRqqt3/assignment_loss‚1.5788639264646918 1loss/FBt1/assignment_loss‚0.14091884312612799 3loss/FBt3/assignment_loss‚0.0063319285691250116 1loss/SRqqt1/detection_loss‚1.1037162878801898 /loss/FBt2/detection_loss‚2.3648262725156894 /loss/FBt4/detection_loss‚1.8902790082825556 2loss/SRqqt2/assignment_loss‚1.2745790967234858 0loss/FBt4/assignment_loss‚0.2540609834217725 /loss/FRt4/detection_loss‚2.6165892248218126 /loss/FBt1/detection_loss‚0.5841165998211021 % _timestamp‚1.7884446173571126e+09 _step‚689 /loss/FRt1/detection_loss‚1.9357547252193879 2loss/SRqqt1/assignment_loss‚1.0122584424735535 2loss/SRqqt4/assignment_loss‚1.6832345672629097 1loss/SRqqt2/detection_loss‚1.6635954109606919 _runtime‚7057.401685664±Ë‘·!‚(:† ÐÿåÔ¨¯¨É gpu.0.gpu‚64.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚96.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WoPD?¸!‚(:´ ÐÿåÔÌ cpu‚25.30134608491605! disk.nvme0n1.in‚ 1755.46484375% disk.nvme0n1.out‚59206.8544921875 proc.cpu.threads‚17" disk.dm-0.out‚59206.8544921875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚59206.8544921875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 257.1484375 disk.sda1.in‚0) disk./.usagePercent‚74.30279164132534 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.sda2.out‚0% disk./.usageGB‚1107.1652908325195 disk.sda3.out‚ 257.1484375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.in‚0! disk.rbd2.out‚581.68408203125 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.373046875$ memory_percent‚89.15219306945801( proc.memory.availableMB‚ 1777.3046875) proc.memory.percent‚16.06571674346924 network.recv‚ 2.551798e+06 disk.sda2.in‚0 network.sent‚ 2.0823169e+07 disk.dm-1.out‚ 257.1484375 disk.rbd3.in‚0# proc.memory.rssMB‚ 2632.20703125Q0)Γ¹!‚(:ˆ ßÿåÔ›¦‘É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚119.045' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.809 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏY´ð6º!‚(:« ßÿåÔ¥³ŠË disk.drbd1777.in‚0 disk.sda.out‚ 257.78125) proc.memory.percent‚16.06571674346924 disk.rbd0.out‚0 disk.dm-0.in‚ 1755.46484375 proc.cpu.threads‚17 disk.sda2.out‚0! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.in‚0 network.recv‚ 2.558959e+06) disk./.usagePercent‚74.29153755012014 disk.rbd1.out‚ 3.373046875 disk.rbd3.out‚0 disk.sda3.out‚ 257.78125 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 257.78125$ memory_percent‚89.15205001831055$ disk.nvme0n1.out‚59209.541015625! disk.dm-0.out‚59209.541015625' proc.memory.availableMB‚ 1777.328125 disk.sda1.in‚0 network.sent‚ 2.0932808e+07 disk.sda2.in‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2632.20703125 cpu‚25.300824859024537 disk.sda.in‚0.5625 disk.sda1.out‚0% disk./.usageGB‚1106.9975967407227 disk.drbd1777.out‚0 disk.rbd1.in‚0! disk.rbd2.out‚583.19580078125& disk.nvme0c0n1.out‚59209.541015625·³ÚÇŸ»!‚(:” îÿåÔ´ÏÃÉ gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚274.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.948600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å´ŽÈ;¼!‚(:° îÿåÔ¤êŽÌ cpu‚25.300323976937882$ disk./.usageGB‚1106.997631072998 disk.rbd2.out‚583.3720703125! disk.nvme0n1.in‚ 1755.46484375$ memory_percent‚89.15205001831055 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.373046875 disk.sda.in‚0.5625 disk.sda.out‚ 258.15234375 disk.rbd3.in‚0$ disk.nvme0n1.out‚59209.857421875 disk.rbd0.out‚0' proc.memory.availableMB‚ 1777.328125 proc.cpu.threads‚17 network.recv‚ 2.56269e+06 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625& disk.nvme0c0n1.out‚59209.857421875 disk.dm-1.out‚ 258.15234375# proc.memory.rssMB‚ 2632.20703125# disk.nvme0c0n1.in‚ 1755.46484375! disk.dm-0.out‚59209.857421875 network.sent‚ 2.093806e+07) disk./.usagePercent‚74.29153985418793 disk.sda3.out‚ 258.15234375 disk.drbd1777.in‚0) proc.memory.percent‚16.06571674346924Þ2E2½!- _wandbruntime‚7096  _runtime‚7096¡ªŸˆÇ¾!Á _step‚690 & lr-AdamW/pg1‚9.755757471138458e-05 & lr-AdamW/pg2‚9.755757471138458e-05 trainer/global_step‚8449 % _timestamp‚1.7884446567345748e+09 _runtime‚7096.779304062èa£®Ì¿!Æ _step‚690 & lr-AdamW/pg1‚9.755757471138458e-05 & lr-AdamW/pg2‚9.755757471138458e-05 trainer/global_step‚8449 % _timestamp‚1.7884446567345748e+09 _runtime‚7096.779304062²­ßF2À!- _wandbruntime‚7096  _runtime‚7096ܵ”(Á!ŠP GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15666032801792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚940006789369856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚804163422284800 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21456070326784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955672848563200 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124419689 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124419683 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21456019736064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚940717108907244 _runtime‚7096.780394346 % _timestamp‚1.7884446567354312e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124419683 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15663432592384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚825619492611584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚940006789369856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚925077023058944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15640085848300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955670221962240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796444 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157396795 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157376761 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157396795 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139167106 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15637487563632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157376761 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80706086 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139166606 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139166606 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21456070326784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955672848563200 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124419689 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124419683 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21456019736064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚940717108907244 _runtime‚7096.780394346 % _timestamp‚1.7884446567354312e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124419683 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15663432592384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚825619492611584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚940006789369856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚925077023058944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15640085848300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955670221962240 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776³ £ø2Ã!- _wandbruntime‚7097  _runtime‚7097QÁØÛÊÄ!”N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚804255782203904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 trainer/global_step‚8449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 % _timestamp‚1.7884446574917133e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955782620610560 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚925183338805248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚7097.537653685 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15665264921088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139182376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80715282 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15667865143808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚940114755466752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157415055 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15665264921088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124433925 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281828914 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚825714374477312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955779993996288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848980 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124433919 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124433925 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15639316906300 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚955782620610560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281828914 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124433919 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚940822655711548 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139182864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚804255810338304 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 trainer/global_step‚8449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 % _timestamp‚1.7884446574917133e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955782620610560 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚925183338805248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚7097.537653685 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15665264921088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139182376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80715282 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15667865143808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚940114755466752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157415055 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15665264921088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124433925 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281828914 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚825714374477312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955779993996288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848980 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124433919 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124433925 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15639316906300 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚955782620610560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281828914 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124433919 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚940822655711548 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139182864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚804255810338304 BÌ!‚(:³ Œ€æÔÇÏË disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 259.09375& proc.memory.availableMB‚ 1691.46875 cpu‚25.310222380234592 network.sent‚ 2.1053169e+07 disk.rbd1.in‚0 disk.rbd2.out‚584.9892578125 disk.sda2.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1755.46484375( disk./.usagePercent‚74.3081701036122 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.521484375 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1755.46484375$ memory_percent‚89.67609405517578 network.recv‚ 2.573786e+06$ disk./.usageGB‚1107.245433807373 disk.rbd2.in‚0 disk.sda.out‚ 259.08984375 disk.sda3.out‚ 259.08984375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2717.79296875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚59337.36279296875 disk.sda2.out‚0& disk.nvme0n1.out‚59337.36279296875 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1755.46484375# disk.dm-0.out‚59337.36279296875 disk.rbd3.out‚0) proc.memory.percent‚16.58809185028076›p°’Í!‚(:‡ ›€æÔ›—ûÈ gpu.0.gpu‚90.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÀ1<Î!‚(:± ›€æÔˆ„ÉË network.sent‚ 2.1059555e+07) disk./.usagePercent‚74.30813733464781 disk.dm-1.out‚ 259.765625 disk.sda.out‚ 259.765625 disk.rbd3.out‚0% disk.nvme0n1.out‚59465.3271484375 disk.rbd0.out‚0 network.recv‚ 2.578142e+06 disk.sda2.in‚0 disk.rbd1.out‚ 3.521484375 disk.sda1.out‚0) proc.memory.availableMB‚ 1691.20703125$ disk./.usageGB‚1107.244945526123 disk.drbd1777.in‚0 disk.rbd2.out‚586.5634765625 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚59465.3271484375 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1755.46484375 disk.sda3.out‚ 259.765625" disk.dm-0.out‚59465.3271484375 disk.sda1.in‚0# proc.memory.rssMB‚ 2717.79296875) proc.memory.percent‚16.58809185028076 cpu‚25.319999784066326 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.rbd3.in‚0$ memory_percent‚89.67769145965576NLMžÏ!‚(:“ ª€æÔÿ©­É gpu.0.gpu‚78.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚37.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Þ`h<Ð!‚(:± ª€æÔ¡ìÔË disk.rbd1.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2717.79296875) proc.memory.percent‚16.58809185028076% disk./.usageGB‚1107.2447547912598 disk.dm-1.out‚ 260.34375 disk.sda.out‚ 260.34375 disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.rbd1.out‚ 3.5537109375 disk.sda2.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1755.46484375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0! disk.rbd2.out‚586.86083984375 cpu‚25.329947767954156 network.recv‚ 2.582031e+06) disk./.usagePercent‚74.30812453427109 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1755.46484375' disk.nvme0c0n1.out‚59465.7802734375 disk.dm-0.in‚ 1755.46484375) proc.memory.availableMB‚ 1691.53515625 network.sent‚ 2.1064783e+07 disk.sda3.out‚ 260.34375 disk.rbd0.in‚0" disk.dm-0.out‚59465.7802734375 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625$ memory_percent‚89.67568874359131 proc.cpu.threads‚17% disk.nvme0n1.out‚59465.7802734375E ’02Ñ!- _wandbruntime‚7159  _runtime‚7159xóÊ­¯&Ò!óT aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15714937118208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15974 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚830008782854656 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 151499264 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚830008631355392 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚808493990401024 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚960847079561728 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157891 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚945132118149120 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 139977179 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚81223398 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23672651776 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚15712986399744 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 283157891 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2649 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚960847079561728 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 133850624 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012864 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚21514792453632 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 158005092 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚150 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15688875351824 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58753629 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚808493972752384 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚945682756656312 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_poÉ+•èÊol.freed‚15686926549176 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚960845104548864 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚21514658603008 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2649 trainer/global_step‚8462 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚945132142443520 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚945132142443520 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 157989118 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464y°†/~*Ó!øT J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 158005092 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1575 _runtime‚7159.715528672 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 125152794 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚152 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 1950718464 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚15714937118208 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 139977027 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚960847079561728 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157891 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚945132118149120 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 139977179 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚81223398 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23672651776 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚15712986399744 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 283157891 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2649 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚960847079561728 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 133850624 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012864 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚21514792453632 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 158005092 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚150 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15688875351824 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58753629 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚808493972752384 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚945682756656312 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚15686926549176 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚960845104548864 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚21514658603008 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2649 trainer/global_step‚8462 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚945132142443520 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚945132142443520 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 157989118 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15714937118208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15974 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚830008782854656 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 151499264 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚830008631355392 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚808493990401024 B2Ô!- _wandbruntime‚7160  _runtime‚7160;[¹ƒ)Õ!ýR cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚945205901031424 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚945755202576464 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚945205876737024 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 125163693 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚151 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚945753230443000 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 158012306 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚808555955106816 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 125163688 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚830071087912960 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948802648[OUà'Ö!‚S JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 283160020 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚830071239412224 _runtime‚7160.727966031 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 157996332 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1575 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚81231328 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1575 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚21515132806144 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 283160020 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 125163688 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 20788341248 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 157996332 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚945205876737024 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15715411321344 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 151499264 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133464 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚960919337339904 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 139989105 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚21515266656768 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 125163693 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚945205901031424 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15715411321344 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚930065831207936 _step‚695 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚960921312352768 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚960919337339904 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23672651776 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884447206827712e+09 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 283175999 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15687399235064 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 133850624 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1575 trainer/global_step‚8462 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 139989258 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2711634944 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚945753230443000 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 158012306 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚808555955106816 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 125163688 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚830071087912960 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948802648 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚945205901031424 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚945755202576464 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚945205876737024 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 125163693 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚151 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.194378063082695 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.531777560710907 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997744083404541 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46186426281929016 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7083333134651184 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.033616747707128525 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5177632570266724 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27581334114074707 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10424237698316574 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26522988080978394 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25986137986183167 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012141293846070766 +particle/sensitivity‚0.3156608045101166 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 epoch‚6 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35408902168273926 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5360596179962158 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4971264600753784 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04391788691282272 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7083333134651184 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07983443886041641 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_1_of_5‚0.9600166082382202 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27668294310569763 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2187907099723816 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5580509901046753 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.022670011967420578 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12452107667922974 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_3_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_1_of_2‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.462905615568161 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22585557401180267 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5251435041427612 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.252525269985199 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21176651120185852 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16128070652484894 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970352649688721 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3236607611179352 particle/accuracy_2_of_5‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5851638913154602 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10279694199562073 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23302467167377472 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5608958601951599 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 jet/accuracy_2_of_7‚1 particle/accuracy_7_of_7‚0.5 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4969135820865631 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19584299623966217 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08926408737897873 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153648376464844 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0008818342466838658 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10639400780200958 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926144480705261 particle/accuracy_4_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 _runtime‚7174.086636682 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23314203321933746 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917290210723877 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25665053725242615 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9892160892486572 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17968444526195526 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 $ _timestamp‚1.788444734040014e+09 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.992141604423523 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5351812243461609 *jet/accuracy_1_of_1‚0.4966031014919281 *jet/accuracy_1_of_3‚0.8718886971473694 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943242073059082 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5255388617515564 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5708727836608887 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45331379771232605 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3378652036190033 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.605555534362793 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9871731400489807 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6095237731933594 ,jet/accuracy_4_of_4‚0.004587306175380945 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12654419243335724 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,jet/accuracy_4_of_5‚0.010522273369133472 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5246233940124512 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03260267525911331 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42365580797195435 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15922300517559052 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2864646911621094 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4127906858921051 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5050633549690247 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.49200206995010376 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5233823657035828 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5884706974029541 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5511189699172974 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.000933706818614155 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19418549537658691 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5385304093360901 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.047484274953603745 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22647693753242493 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41912201046943665 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6978000998497009 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42275163531303406 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08189655095338821 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18965516984462738 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5225119590759277 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1579885482788086 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5216690897941589 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_6_of_6‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9958387613296509 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4916794002056122 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3532157242298126 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49382179975509644 +jet/accuracy_2_of_4‚0.37258899211883545 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_2_of_7‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24143525958061218 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_7_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.393833190202713 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3018694221973419 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980308413505554 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895081520080566 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6577152013778687 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1657785177230835 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13061568140983582 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_3_of_4‚0.06560567766427994 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7549262642860413 6validation_average_jet_accuracy‚0.4445914030075073 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5694444179534912 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1604515165090561 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020443666726350784 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5147238969802856 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4996470808982849 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29242947697639465 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6412516832351685 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19687741994857788 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 jet/accuracy_6_of_6‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18974915146827698 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5253592133522034 particle/accuracy_1_of_1‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.010457516647875309 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0714285746216774 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28149548172950745 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_5_of_5‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.424408495426178 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1328357309103012 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20262718200683594 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.586904764175415 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08493880182504654 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14930085837841034 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.020653819665312767 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1865386962890625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9820402264595032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5877634882926941 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393940567970276 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 +jet/accuracy_3_of_6‚0.23076923191547394 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9891563057899475 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 'particle/f_score‚0.4001409113407135 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.98097163438797 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9277396202087402 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.547724723815918 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9866712093353271 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5040714144706726 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.034805428236722946 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5056251287460327 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4947381317615509 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008282922208309174 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +particle/specificity‚0.9516627192497253 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35277777910232544 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904078841209412 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18654969334602356 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.215277761220932 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18597659468650818 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17783969640731812 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +jet/accuracy_2_of_2‚0.11773792654275894 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.037079598754644394 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_3_of_3‚0.02784247323870659 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 _step‚696 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4238710105419159 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 jet/accuracy_5_of_6‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911067485809326 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8523374795913696 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6066656112670898 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5493636727333069 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27407410740852356 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9929183721542358 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1883593499660492 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4506579041481018 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11377865076065063 particle/accuracy_4_of_5‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01529385894536972 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3586805760860443 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36416780948638916 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.272560179233551 particle/acc1õÎuracy_3_of_4‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 *jet/accuracy_2_of_6‚0.5512820482254028 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1865386962890625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5621284246444702 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5153290033340454 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5195045471191406 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4940650463104248 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20923061668872833 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13282421231269836 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2793099880218506 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37451499700546265 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.413943350315094 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 *loss/val_total_loss‚40.331628088312854 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5172294974327087 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1999100148677826 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2275083065032959 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15047885477542877 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 (jet/accuracy_2_of_3‚0.26274374127388 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868060946464539 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4508620500564575 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.46456456184387207 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5553562641143799 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1622222363948822 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.567072868347168 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 jet/accuracy_5_of_7‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5255388617515564 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13834817707538605 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08988095074892044 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5380053520202637 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1677754521369934 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9869509935379028 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.613095223903656 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2635135054588318 particle/accuracy_3_of_7‚1 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.577346682548523 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.45746928453445435 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6079404950141907 jet/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 trainer/global_step‚8462 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16354364156723022 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.48215487599372864 particle/accuracy_3_of_6‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.42712417244911194 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5089696049690247 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4864628314971924 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1419753134250641 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273éÉ$dÛ!ö§ ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5580509901046753 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.022670011967420578 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.05319149047136307 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12452107667922974 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_3_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_1_of_2‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.462905615568161 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22585557401180267 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5251435041427612 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.252525269985199 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21176651120185852 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16128070652484894 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970352649688721 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3236607611179352 particle/accuracy_2_of_5‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5851638913154602 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10279694199562073 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23302467167377472 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5608958601951599 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 jet/accuracy_2_of_7‚1 particle/accuracy_7_of_7‚0.5 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4969135820865631 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19584299623966217 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08926408737897873 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153648376464844 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0008818342466838658 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10639400780200958 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926144480705261 particle/accuracy_4_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 _runtime‚7174.086636682 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23314203321933746 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917290210723877 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25665053725242615 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9892160892486572 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17968444526195526 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 $ _timestamp‚1.788444734040014e+09 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.992141604423523 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5351812243461609 *jet/accuracy_1_of_1‚0.4966031014919281 *jet/accuracy_1_of_3‚0.8718886971473694 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943242073059082 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5255388617515564 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5708727836608887 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45331379771232605 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3378652036190033 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.605555534362793 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9871731400489807 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6095237731933594 ,jet/accuracy_4_of_4‚0.004587306175380945 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12654419243335724 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,jet/accuracy_4_of_5‚0.010522273369133472 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5246233940124512 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03260267525911331 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42365580797195435 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15922300517559052 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2864646911621094 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4127906858921051 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5050633549690247 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.49200206995010376 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5233823657035828 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5884706974029541 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5511189699172974 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.000933706818614155 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19418549537658691 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5385304093360901 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.047484274953603745 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22647693753242493 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41912201046943665 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6978000998497009 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42275163531303406 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08189655095338821 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18965516984462738 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5225119590759277 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1579885482788086 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5216690897941589 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_6_of_6‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9958387613296509 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4916794002056122 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3532157242298126 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49382179975509644 +jet/accuracy_2_of_4‚0.37258899211883545 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_2_of_7‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24143525958061218 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_7_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.393833190202713 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3018694221973419 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980308413505554 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895081520080566 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6577152013778687 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1657785177230835 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13061568140983582 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_3_of_4‚0.06560567766427994 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7549262642860413 6validation_average_jet_accuracy‚0.4445914030075073 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5694444179534912 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1604515165090561 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020443666726350784 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5147238969802856 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4996470808982849 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29242947697639465 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6412516832351685 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19687741994857788 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333015441895 jet/accuracy_6_of_6‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18974915146827698 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5253592133522034 particle/accuracy_1_of_1‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.010457516647875309 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0714285746216774 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28149548172950745 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_5_of_5‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.424408495426178 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1328357309103012 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20262718200683594 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.586904764175415 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08493880182504654 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14930085837841034 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.020653819665312767 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1865386962890625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9820402264595032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5877634882926941 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393940567970276 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 +jet/accuracy_3_of_6‚0.23076923191547394 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9891563057899475 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 'particle/f_score‚0.4001409113407135 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.98097163438797 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9277396202087402 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.547724723815918 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9866712093353271 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5040714144706726 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.034805428236722946 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5056251287460327 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4947381317615509 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008282922208309174 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +particle/specificity‚0.9516627192497253 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35277777910232544 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904078841209412 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18654969334602356 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.215277761220932 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18597659468650818 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17783969640731812 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +jet/accuracy_2_of_2‚0.11773792654275894 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.037079598754644394 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_3_of_3‚0.02784247323870659 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 _step‚696 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4238710105419159 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 jet/accuracy_5_of_6‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911067485809326 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8523374795913696 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6066656112670898 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5493636727333069 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27407410740852356 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt1SRqqt1FBt/FB£fö Ù/t_purity‚0.9929183721542358 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1883593499660492 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4506579041481018 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11377865076065063 particle/accuracy_4_of_5‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01529385894536972 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3586805760860443 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36416780948638916 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.272560179233551 particle/accuracy_3_of_4‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 *jet/accuracy_2_of_6‚0.5512820482254028 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1865386962890625 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5621284246444702 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5153290033340454 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5195045471191406 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4940650463104248 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20923061668872833 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13282421231269836 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2793099880218506 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37451499700546265 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.413943350315094 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 *loss/val_total_loss‚40.331628088312854 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5172294974327087 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1999100148677826 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2275083065032959 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15047885477542877 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 (jet/accuracy_2_of_3‚0.26274374127388 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868060946464539 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4508620500564575 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.46456456184387207 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5553562641143799 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1622222363948822 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.567072868347168 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 jet/accuracy_5_of_7‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5255388617515564 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13834817707538605 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08988095074892044 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5380053520202637 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1677754521369934 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9869509935379028 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.613095223903656 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2635135054588318 particle/accuracy_3_of_7‚1 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.577346682548523 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.45746928453445435 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6079404950141907 jet/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 trainer/global_step‚8462 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16354364156723022 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.48215487599372864 particle/accuracy_3_of_6‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.42712417244911194 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5089696049690247 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4864628314971924 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1419753134250641 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43651899695396423 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9931970834732056 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03186318650841713 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.594093382358551 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.194378063082695 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.531777560710907 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997744083404541 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.46186426281929016 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7083333134651184 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.033616747707128525 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5177632570266724 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27581334114074707 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10424237698316574 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26522988080978394 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25986137986183167 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012141293846070766 +particle/sensitivity‚0.3156608045101166 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 epoch‚6 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35408902168273926 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5360596179962158 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4971264600753784 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04391788691282272 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7083333134651184 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07983443886041641 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_1_of_5‚0.9600166082382202 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27668294310569763 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2187907099723816 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1¸x&‘Ü!‚(:† È€æÔ…Œ—É gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚33.0 gpu.0.powerWatts‚76.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.3326 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ÐãFÝ!‚(:» È€æÔá²ÞË network.recv‚ 2.600544e+06! disk.rbd2.out‚590.04833984375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda3.out‚ 261.3671875) proc.memory.availableMB‚ 3746.12890625 proc.cpu.threads‚17 network.sent‚ 2.1183765e+07% disk./.usageGB‚1107.2532005310059 disk.drbd1777.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚64084.27392578125 disk.dm-1.out‚ 261.3671875# proc.memory.rssMB‚ 2725.13671875 cpu‚25.341539301503232 disk.sda2.in‚0( disk.nvme0c0n1.out‚64084.27392578125 disk.dm-0.in‚ 1755.46484375$ memory_percent‚77.13544368743896* proc.memory.percent‚16.632914543151855 disk.drbd1777.in‚0& disk.nvme0n1.out‚64084.27392578125# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.out‚ 3.61669921875 disk.sda3.in‚0.5625( disk./.usagePercent‚74.3086913349519 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 261.3671875 disk.sda1.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.in‚0 disk.dm-1.in‚0.5625LY“Ó’Þ!‚(:‡ ×€æÔäاÉ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦]Q>ß!‚(:³ ×€æÔàåÁË* proc.memory.percent‚16.632914543151855 network.recv‚ 2.607661e+06 disk.sda3.out‚ 261.703125 disk.rbd0.out‚0 disk.dm-0.in‚ 1755.46484375 disk.dm-1.out‚ 261.703125 disk.drbd1777.out‚0( disk./.usagePercent‚74.3087294800745 disk.sda.out‚ 261.703125 disk.sda2.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 3745.984375! disk.rbd2.out‚590.04833984375 disk.sda1.in‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.61669921875$ memory_percent‚77.13632583618164 cpu‚25.340935813081806% disk./.usageGB‚1107.2537689208984 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1755.46484375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚64882.3544921875" disk.dm-0.out‚64882.3544921875 disk.sda2.out‚0! disk.nvme0n1.in‚ 1755.46484375% disk.nvme0n1.out‚64882.3544921875# proc.memory.rssMB‚ 2725.13671875 network.sent‚ 2.1309163e+07 disk.sda.in‚0.5625Û72à!- _wandbruntime‚7213  _runtime‚7213ìÛÆá!À _runtime‚7213.509933975 _step‚697 & lr-AdamW/pg1‚9.513968675374312e-05 & lr-AdamW/pg2‚9.513968675374312e-05 trainer/global_step‚8499 $ _timestamp‚1.788444773465201e+097ì)xËâ!Å trainer/global_step‚8499 $ _timestamp‚1.788444773465201e+09 _runtime‚7213.509933975 _step‚697 & lr-AdamW/pg1‚9.513968675374312e-05 & lr-AdamW/pg2‚9.513968675374312e-05¹b¬‰}2ã!- _wandbruntime‚7213  _runtime‚7213lD±'(ä!‰P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15784955229696 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125817811 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81665466 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚834300608321024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15758764606562 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59054000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950565514014174 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59053554 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚834300535887360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15761362891230 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 _step‚698 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125817805 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚950050587964416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140719020 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 _runtime‚7213.510889034 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 $ _timestamp‚1.788444773466046e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15784955229696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚812687728847872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158748232 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140719469 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚812687700713472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158768266 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚965838143403520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950562890301538 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125817811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚934804125694976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81665469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚950050561572864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158748232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21612835173888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284566037 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚934804151122944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚965838143403520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965835516802560 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15787555439104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15787555439104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125817805 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21612879473152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚8499 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚950050561572864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚965835516802560 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284566037 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚934804151122944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚965838143403520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965835516802560 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15787555439104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15787555439104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125817805 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21612879473152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚8499 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚950050561572864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚965835516802560 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140719020 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 _runtime‚7213.510889034 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 $ _timestamp‚1.788444773466046e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15784955229696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚812687728847872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158748232 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140719469 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚812687700713472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158768266 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚965838143403520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560ºÎÝ2æ!- _wandbruntime‚7214  _runtime‚7214µUb'ç!”N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140734931 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59060269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15786787558400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15760593949230 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15786787558400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21615416892416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158786526 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125832047 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125832041 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834395533793792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7214.269187456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚950158501278208 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965945288836608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚965947915450880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚834395461373440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚950671060818478 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21615372606464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81674665 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚812780116901376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81674662 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158766466 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884447742237432e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140735345 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125832047 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125832041 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 trainer/global_step‚8499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚965947915450880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚950158527669760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158786526 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15789387781120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚411 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59060680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚950673684531322 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158766466 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15789387781120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚950158527669760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934910492297216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15763192234106 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400Ë Q'è!™N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚950673684531322 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158766466 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15789387781120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚950158527669760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚934910492297216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15763192234106 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140734931 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59060269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15786787558400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15760593949230 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15786787558400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21615416892416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158786526 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125832047 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125832041 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834395533793792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7214.269187456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚950158501278208 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965945288836608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚965947915450880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚834395461373440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚950671060818478 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21615372606464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81674665 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚812780116901376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81674662 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158766466 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884447742237432e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140735345 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125832047 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125832041 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 trainer/global_step‚8499 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚965947915450880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284618573 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚950158527669760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158786526 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15789387781120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚411 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59060680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376»+ªf2é!- _wandbruntime‚7214  _runtime‚7214NÀض^ê!Ø .loss/FBt3/detection_loss‚2.157151257991791 1loss/FBt1/assignment_loss‚0.08755881742616434 /loss/FRt4/detection_loss‚2.4811583757400513 /loss/FBt4/detection_loss‚2.3740241646766664 trainer/global_step‚8499 0loss/FRt1/assignment_loss‚1.0154696914056938 0loss/FRt2/assignment_loss‚2.1704054384127907 0loss/FBt2/assignment_loss‚0.3120071699210842 /loss/FRt1/detection_loss‚1.3851180511216323 /loss/FRt3/detection_loss‚2.3744933671915045 /loss/FBt2/detection_loss‚2.3641170603888377 2loss/SRqqt2/assignment_loss‚1.0176247013908513 %loss/total_loss‚36.52485251585698 epoch‚7 _step‚700 0loss/FRt4/assignment_loss‚2.4347131187851363 1loss/FBt4/assignment_loss‚0.43137381701089905 1loss/SRqqt2/detection_loss‚1.8411245970231183 0loss/SRqqt3/detection_loss‚2.298754088340267 /loss/FRt3/assignment_loss‚2.856379788975383 2loss/SRqqt1/assignment_loss‚0.9136841443167734 .loss/FRt2/detection_loss‚2.258053458903147 /loss/FBt1/detection_loss‚0.5071297412068816 % _timestamp‚1.7884447742261133e+09 2loss/SRqqt3/assignment_loss‚1.1608895803651502 1loss/SRqqt1/detection_loss‚1.2345897428624864 2loss/SRqqt4/assignment_loss‚0.8693885624608291 1loss/FBt3/assignment_loss‚0.23266230709850788 1loss/SRqqt4/detection_loss‚1.7469814728413309 _runtime‚7214.270752755%¯ácë!Ý 1loss/FBt1/assignment_loss‚0.08755881742616434 /loss/FRt4/detection_loss‚2.4811583757400513 /loss/FBt4/detection_loss‚2.3740241646766664 trainer/global_step‚8499 0loss/FRt1/assignment_loss‚1.0154696914056938 0loss/FRt2/assignment_loss‚2.1704054384127907 0loss/FBt2/assignment_loss‚0.3120071699210842 /loss/FRt1/detection_loss‚1.3851180511216323 /loss/FRt3/detection_loss‚2.3744933671915045 /loss/FBt2/detection_loss‚2.3641170603888377 2loss/SRqqt2/assignment_loss‚1.0176247013908513 %loss/total_loss‚36.52485251585698 epoch‚7 _step‚700 0loss/FRt4/assignment_loss‚2.4347131187851363 1loss/FBt4/assignment_loss‚0.43137381701089905 1loss/SRqqt2/detection_loss‚1.8411245970231183 0loss/SRqqt3/detection_loss‚2.298754088340267 /loss/FRt3/assignment_loss‚2.856379788975383 2loss/SRqqt1/assignment_loss‚0.9136841443167734 .loss/FRt2/detection_loss‚2.258053458903147 /loss/FBt1/detection_loss‚0.5071297412068816 % _timestamp‚1.7884447742261133e+09 2loss/SRqqt3/assignment_loss‚1.1608895803651502 1loss/SRqqt1/detection_loss‚1.2345897428624864 2loss/SRqqt4/assignment_loss‚0.8693885624608291 1loss/FBt3/assignment_loss‚0.23266230709850788 1loss/SRqqt4/detection_loss‚1.7469814728413309 _runtime‚7214.270752755 .loss/FBt3/detection_loss‚2.157151257991791¼ïöŸÐžì!‚(:“ æ€æÔ¼¡É gpu.0.gpu‚69.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚244.313' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚48.86259999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a|9&í!‚(:› æ€æÔô½×Ë% disk./.usageGB‚1107.2538261413574 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda2.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1755.46484375' disk.nvme0c0n1.out‚64882.6396484375$ memory_percent‚77.13754177093506) disk./.usagePercent‚74.30873332018751 disk.drbd1777.out‚0 disk.sda.out‚262 disk.sda2.in‚0" disk.dm-0.out‚64882.6396484375 disk.dm-1.in‚0.5625* proc.memory.percent‚16.632914543151855 cpu‚25.340297436016815 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚64882.6396484375 disk.dm-1.out‚262) proc.memory.availableMB‚ 3745.78515625# proc.memory.rssMB‚ 2725.13671875 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1755.46484375 disk.sda.in‚0.5625 disk.rbd2.out‚ 591.751953125 disk.sda3.in‚0.5625 disk.sda3.out‚262 network.sent‚ 2.1314377e+07 network.recv‚ 2.611434e+06 disk.sda1.in‚0! disk.nvme0n1.in‚ 1755.46484375z–ð¼“î!‚(:ˆ õ€æÔª¥‡É gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚247.845' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.569 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â­D<ï!‚(:± õ€æÔ¢øèÊ disk.rbd3.in‚0$ disk.nvme0n1.out‚64883.361328125 disk.rbd0.in‚0! disk.dm-0.out‚64883.361328125 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.6796875 cpu‚25.339738146781194 network.recv‚ 2.617921e+06% disk./.usageGB‚1107.2538566589355! disk.nvme0n1.in‚ 1755.46484375 disk.sda.out‚ 262.56640625 disk.sda1.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1755.46484375& disk.nvme0c0n1.out‚64883.361328125( disk./.usagePercent‚74.3087353682478 disk.sda3.out‚ 262.56640625 disk.dm-1.out‚ 262.56640625 disk.rbd2.in‚0 disk.sda2.in‚0$ memory_percent‚77.13532447814941 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.sda1.in‚0# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17 network.sent‚ 2.1423872e+07 disk.dm-0.in‚ 1755.46484375 disk.rbd2.out‚ 591.751953125 disk.sda2.out‚0( proc.memory.availableMB‚ 3746.1484375 disk.sda.in‚0.5625 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚16.632914543151855ht¿žð!‚(:“ „æÔÁ°µÉ gpu.0.gpu‚90.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚172.531' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.50620000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­=ñ!‚(:² „æÔüѳË disk.sda.in‚0.5625* proc.memory.percent‚16.632914543151855$ disk.nvme0n1.out‚64883.642578125 disk.rbd0.out‚0 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-1.out‚ 262.99609375 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 593.490234375 disk.sda1.in‚0) proc.memory.availableMB‚ 3745.98828125 disk.rbd3.out‚0 network.sent‚ 2.1429616e+07& disk.nvme0c0n1.out‚64883.642578125 disk.sda1.out‚0# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17! disk.dm-0.out‚64883.642578125! disk.nvme0n1.in‚ 1755.46484375) disk./.usagePercent‚74.30873408821012% disk./.usageGB‚1107.2538375854492 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 262.99609375 disk.sda2.in‚0$ memory_percent‚77.13630199432373 cpu‚25.339136234156875 disk.sda3.out‚ 262.99609375 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 network.recv‚ 2.62215e+06 disk.dm-0.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.in‚0³žû2ò!- _wandbruntime‚7253  _runtime‚7253e4  Çó!Á & lr-AdamW/pg2‚9.274235050617121e-05 trainer/global_step‚8549 % _timestamp‚1.7884448133153052e+09 _runtime‚7253.360046699 _step‚701 & lr-AdamW/pg1‚9.274235050617121e-05Ë·(õÌô!Æ & lr-AdamW/pg1‚9.274235050617121e-05 & lr-AdamW/pg2‚9.274235050617121e-05 trainer/global_step‚8549 % _timestamp‚1.7884448133153052e+09 _runtime‚7253.360046699 _step‚701½H_‘2õ!- _wandbruntime‚7253  _runtime‚7253’t 4Úö!ŠP dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21739366868992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82125266 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚955447546840064 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7253.360987692 SDDeviceStatsMonitor.on_train_batch_start/actiæ u6#ve_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159683318 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59391282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141516548 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚940121184408576 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚955447546840064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126529605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚839046498392064 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15852843615838 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚955971429739746 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15879186288128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚955974053452382 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚940121209836544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286212929 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126529605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15876586078720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚971324132918784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21739320472576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15876586078720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82125269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚971326759519744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15850245331170 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚971326759519744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚971324132918784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚839046423861248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15850245331170 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚971326759519744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚971324132918784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚839046423861248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7253.360987692 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159683318 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59391282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141516548 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚940121184408576 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚955447546840064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126529605 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚839046498392064 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15852843615838 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚955971429739746 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15879186288128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚955974053452382 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚940121209836544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286212929 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126529605 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15876586078720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚971324132918784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21739320472576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15876586078720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776¾ú˜ý2ø!- _wandbruntime‚7254  _runtime‚72543HpÑ'ù!“N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚839141392223232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159681518 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286225359 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15878418407424 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126543841 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21741872646656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21741826263552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788444814070101e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚955555512936960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82134465 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚817399491442176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 trainer/global_step‚8549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚7254.115332045 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚971433904952832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15852074673838 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15881018630144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚940227525582848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159701578 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚956082223969530 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159701578 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59398360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚940227551010816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15854672958714 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126543841 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126543847 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚955555486545408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚971436531567104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141532393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245425 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159701578 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚956082223969530 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159701578 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59398360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚940227551010816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15854672958714 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126543841 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126543847 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚955555486545408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚971436531567104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141532393 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245425 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159681518 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286225359 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15878418407424 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126543841 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21741872646656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21741826263552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788444814070101e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552¿ŽïdM2û!- _wandbruntime‚7254  _runtime‚7254¥Ÿ·Yü!Ó 1loss/SRqqt4/detection_loss‚1.7546125403046609 2loss/SRqqt2/assignment_loss‚1.5567494000265882 2loss/SRqqt4/assignment_loss‚1.2067074199765921 /loss/FRt3/detection_loss‚2.3934794008855533 epoch‚7 0loss/FRt4/assignment_loss‚2.8694672841917384 2loss/SRqqt1/assignment_loss‚0.8583428150350301 % _timestamp‚1.7884448140719018e+09 _step‚704 2loss/SRqqt3/assignment_loss‚1.4922988244465418 1loss/FBt2/assignment_loss‚0.17769747924131268 /loss/FRt1/detection_loss‚1.9898780808699401 0loss/SRqqt3/detection_loss‚2.110891526937485 1loss/FBt1/assignment_loss‚0.06269730715605094 .loss/FRt4/detection_loss‚2.425856992941011 1loss/SRqqt1/detection_loss‚1.0152378403581679 1loss/SRqqt2/detection_loss‚2.0660455487668514 _runtime‚7254.116444661 1loss/FBt4/assignment_loss‚0.26919314680465806 ,loss/FBt1/detection_loss‚0.5086426157241 -loss/FBt2/detection_loss‚2.04781613441614 .loss/FBt3/detection_loss‚2.645297296345234 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚1.1256793615492908 0loss/FRt3/assignment_loss‚3.3096358589828014 /loss/FRt2/detection_loss‚2.3032824102234333 &loss/total_loss‚39.605547591981704 .loss/FBt4/detection_loss‚2.868024597565333 /loss/FRt2/assignment_loss‚2.399975207257778 0loss/FBt3/assignment_loss‚0.1480385019754067,31C^ý!Ø _step‚704 2loss/SRqqt3/assignment_loss‚1.4922988244465418 1loss/FBt2/assignment_loss‚0.17769747924131268 /loss/FRt1/detection_loss‚1.9898780808699401 0loss/SRqqt3/detection_loss‚2.110891526937485 1loss/FBt1/assignment_loss‚0.06269730715605094 .loss/FRt4/detection_loss‚2.425856992941011 1loss/SRqqt1/detection_loss‚1.0152378403581679 1loss/SRqqt2/detection_loss‚2.0660455487668514 _runtime‚7254.116444661 1loss/FBt4/assignment_loss‚0.26919314680465806 ,loss/FBt1/detection_loss‚0.5086426157241 -loss/FBt2/detection_loss‚2.04781613441614 .loss/FBt3/detection_loss‚2.645297296345234 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚1.1256793615492908 0loss/FRt3/assignment_loss‚3.3096358589828014 /loss/FRt2/detection_loss‚2.3032824102234333 &loss/total_loss‚39.605547591981704 .loss/FBt4/detection_loss‚2.868024597565333 /loss/FRt2/assignment_loss‚2.399975207257778 0loss/FBt3/assignment_loss‚0.1480385019754067 1loss/SRqqt4/detection_loss‚1.7546125403046609 2loss/SRqqt2/assignment_loss‚1.5567494000265882 2loss/SRqqt4/assignment_loss‚1.2067074199765921 /loss/FRt3/detection_loss‚2.3934794008855533 epoch‚7 0loss/FRt4/assignment_loss‚2.8694672841917384 2loss/SRqqt1/assignment_loss‚0.8583428150350301 % _timestamp‚1.7884448140719018e+09ÀÇþ!‚(:’ “æÔŸö•É gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚272.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.464800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+è©Aÿ!‚(:¶ “æÔƒÐÏË disk.drbd1777.out‚0 disk.sda.out‚ 263.49609375 disk.sda3.in‚0.5625$ memory_percent‚77.14102268218994 cpu‚25.338539524266288) disk./.usagePercent‚74.31713369540854% disk./.usageGB‚1107.3789978027344 disk.dm-1.out‚ 263.49609375 disk.sda1.in‚0 disk.sda3.out‚ 263.49609375 network.recv‚ 2.626863e+06 disk.rbd2.out‚ 593.490234375 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 3745.21484375 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.sda.in‚0.5625% disk.nvme0n1.out‚64981.8330078125# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375 disk.rbd1.out‚ 3.6796875 disk.rbd3.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17" disk.dm-0.out‚64981.8330078125 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.632914543151855 network.sent‚ 2.1436063e+07! disk.nvme0n1.in‚ 1755.46484375' disk.nvme0c0n1.out‚64981.8330078125 disk.rbd2.in‚0GþçΔ€"‚(:‰ ¢æÔÒðãÈ gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚270.219' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä//±B"‚(:· ¢æÔÓ¢×Ë disk.sda.out‚ 263.76953125 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.632914543151855 network.recv‚ 2.638903e+06% disk./.usageGB‚1107.3789024353027" disk.dm-0.out‚64982.2568359375 disk.sda3.out‚ 263.76953125$ memory_percent‚77.13854312896729 cpu‚25.33791218255971 network.sent‚ 2.1547534e+07 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1755.46484375% disk.nvme0n1.out‚64982.2568359375# disk.nvme0c0n1.in‚ 1755.46484375) proc.memory.availableMB‚ 3745.62109375 proc.cpu.threads‚17 disk.dm-0.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 263.76953125 disk.rbd2.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚64982.2568359375# proc.memory.rssMB‚ 2725.13671875) disk./.usagePercent‚74.31712729522019 disk.sda.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.6796875! disk.rbd2.out‚595.11962890625;ˆ±”‚"‚(:‰ ±æÔæ·ŠÉ gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚253.566' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0víÚ<ƒ"‚(:± ±æÔŽ”“Ì& disk.nvme0n1.out‚64982.84326171875 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.drbd1777.out‚0( disk./.usagePercent‚74.3254424199323 disk.rbd0.out‚0 disk.sda.out‚ 264.3515625 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚88.57498168945312 proc.cpu.threads‚17 disk.sda2.out‚0# proc.memory.rssMB‚ 2725.13671875# disk.dm-0.out‚64982.84326171875 disk.dm-1.out‚ 264.3515625! disk.rbd2.out‚596.63134765625 disk.sda1.in‚0* proc.memory.percent‚16.632914543151855 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda.in‚0.5625 cpu‚25.33738688419025 network.recv‚ 2.642711e+06! disk.nvme0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375$ proc.memory.availableMB‚1871.875 disk.rbd3.out‚0 disk.sda3.out‚ 264.3515625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.in‚0 disk.rbd1.out‚ 3.6796875 network.sent‚ 2.1552794e+07% disk./.usageGB‚1107.5028038024902( disk.nvme0c0n1.out‚64982.84326171875x)2„"- _wandbruntime‚7293  _runtime‚7293M/6ýÇ…"Á & lr-AdamW/pg1‚9.036606556994968e-05 & lr-AdamW/pg2‚9.036606556994968e-05 trainer/global_step‚8599 % _timestamp‚1.7884448531818755e+09 _runtime‚7293.226616738 _step‚705ÆD̆"Æ trainer/global_step‚8599 % _timestamp‚1.7884448531818755e+09 _runtime‚7293.226616738 _step‚705 & lr-AdamW/pg1‚9.036606556994968e-05 & lr-AdamW/pg2‚9.036606556994968e-05Á{iác2‡"- _wandbruntime‚7293  _runtime‚7293‡â“H(ˆ"ŠP [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15941726055778 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15970817137152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127241405 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚843791827580928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚960844558498816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160578336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160578336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59729259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21865370010624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚976815375635968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚976815375635968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15968216927744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚960844532107264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15944324340446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160598370 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚976812749035008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚961382592890590 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚843791904208896 _runtime‚7293.227442293 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819741 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945438243122176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142314325 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59729259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21865370010624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚976815375635968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚976815375635968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15968216927744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚960844532107264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15944324340446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160598370 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚976812749035008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚961382592890590 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚843791904208896 _runtime‚7293.227442293 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819741 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚945438243122176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142314325 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚976922521069056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160616630 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160596570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59736511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127255641 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚960952471812608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59736055 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚976925147683328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚707 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788444853928125e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚976925147683328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160596570 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82594262 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15943555398446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127255647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15972649479168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287872277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚960952498204160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚961490763407738 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚960952471812608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15972649479168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127255641 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚960952498204160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15970049256448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142330317 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82594265 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127255647 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852211 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211û3!É'Œ"˜N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15946153683322 trainer/global_step‚8599 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21867826689536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚976922521069056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160616630 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160596570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59736511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127255641 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚960952471812608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59736055 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚976925147683328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚707 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788444853928125e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚976925147683328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160596570 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82594262 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15943555398446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127255647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15972649479168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287872277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚960952498204160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚961490763407738 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚960952471812608 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15972649479168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127255641 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚960952498204160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15970049256448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142330317 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82594265 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127255647 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287852211 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 P–ÍÁ˜"ˆP UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚966788508616162 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127953205 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60067577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚826545908739072 _step‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚826545936873472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127953211 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982301365151232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161513422 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16035805065054 $ _timestamp‚1.788444893048564e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21991217224704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16059847776768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16059847776768 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143112443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83044869 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚966791132328798 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚982303991752192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚966241517374464 \HDeviceStatsMonitor.on_train_^׈Mbatch_start/allocated_bytes.large_pool.freed‚966241517374464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16062447986176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161513422 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446593 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚966241543766016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161493388 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚950755327263744 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚966241543766016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127953211 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚848537154098176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143112898 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21991170828288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16033206780386 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚848537079567360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚950755301835776 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982303991752192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83044866 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034`î—v(™"P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚966788508616162 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127953205 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60067577 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚826545908739072 _step‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚826545936873472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127953211 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚982301365151232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161513422 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16035805065054 $ _timestamp‚1.788444893048564e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21991217224704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16059847776768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16059847776768 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143112443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83044869 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚966791132328798 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚982303991752192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚966241517374464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚966241517374464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16062447986176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161513422 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289446593 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚966241543766016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161493388 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚950755327263744 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚966241543766016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127953211 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚848537154098176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143112898 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21991170828288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16033206780386 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚848537079567360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚950755301835776 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚982303991752192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83044866 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124Æò¯!¨2š"- _wandbruntime‚7333  _runtime‚7333Ë1¨à'›"”N EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127967441 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127967441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83054062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848632062640640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884448938013976e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83054065 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143128358 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127967447 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚966349483471360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161531682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚982411137185280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16037634407930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚982413763799552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16064280328192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚826638324926976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21993691330560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚966899302845946 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16035036123054 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16064280328192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚966896679133102 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚711 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143128792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚848631988123136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _runtime‚7333.846616609 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60074727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161511622 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚982413763799552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479063 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161511622 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60074296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161531682 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16061680105472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127967447 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ÂÔ1œ"™N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479063 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161511622 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60074296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161531682 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16061680105472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127967447 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127967441 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127967441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83054062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848632062640640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884448938013976e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83054065 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143128358 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127967447 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚966349483471360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161531682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pÜ‹îool.current‚ 3303014400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚982411137185280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16037634407930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚982413763799552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16064280328192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚826638324926976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21993691330560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚966899302845946 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16035036123054 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16064280328192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚966896679133102 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚711 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143128792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚848631988123136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _runtime‚7333.846616609 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60074727 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161511622 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚982413763799552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ç nK²2"- _wandbruntime‚7333  _runtime‚7333Åb«—Yž"Ó /loss/FRt1/assignment_loss‚1.044450758645932 2loss/SRqqt1/assignment_loss‚0.8849427802946235 _runtime‚7333.847773109 _step‚712 .loss/FRt2/detection_loss‚2.404434433070625 /loss/FRt3/detection_loss‚2.3671817413085496 1loss/SRqqt3/detection_loss‚2.5696630573272703 0loss/SRqqt4/detection_loss‚1.557447925083718 .loss/FBt2/detection_loss‚1.775252918402354 %loss/total_loss‚38.50841282347636 epoch‚7 trainer/global_step‚8649 /loss/FRt2/assignment_loss‚2.185222664984261 0loss/FBt1/assignment_loss‚0.1569481957106354 /loss/FRt1/detection_loss‚1.5446644041221589 /loss/FRt4/detection_loss‚2.8157617458275386 0loss/SRqqt1/detection_loss‚1.043368877751417 2loss/SRqqt4/assignment_loss‚1.3611476871548063 /loss/FBt4/detection_loss‚2.5812970995903015 0loss/SRqqt3/assignment_loss‚1.60194697201252 /loss/FBt3/detection_loss‚2.5217370646340505 % _timestamp‚1.7884448938031664e+09 0loss/FRt3/assignment_loss‚3.0204401329944006 /loss/FRt4/assignment_loss‚2.865476514611925 2loss/SRqqt2/assignment_loss‚1.3910012413230208 1loss/FBt2/assignment_loss‚0.12219505965994257 1loss/FBt3/assignment_loss‚0.09814379103564923 1loss/FBt4/assignment_loss‚0.23345425797378994 1loss/SRqqt2/detection_loss‚1.7971079837944772 /loss/FBt1/detection_loss‚0.5651255161624009FÆw^Ÿ"Ø /loss/FBt1/detection_loss‚0.5651255161624009 /loss/FRt1/assignment_loss‚1.044450758645932 2loss/SRqqt1/assignment_loss‚0.8849427802946235 _runtime‚7333.847773109 _step‚712 .loss/FRt2/detection_loss‚2.404434433070625 /loss/FRt3/detection_loss‚2.3671817413085496 1loss/SRqqt3/detection_loss‚2.5696630573272703 0loss/SRqqt4/detection_loss‚1.557447925083718 .loss/FBt2/detection_loss‚1.775252918402354 %loss/total_loss‚38.50841282347636 epoch‚7 trainer/global_step‚8649 /loss/FRt2/assignment_loss‚2.185222664984261 0loss/FBt1/assignment_loss‚0.1569481957106354 /loss/FRt1/detection_loss‚1.5446644041221589 /loss/FRt4/detection_loss‚2.8157617458275386 0loss/SRqqt1/detection_loss‚1.043368877751417 2loss/SRqqt4/assignment_loss‚1.3611476871548063 /loss/FBt4/detection_loss‚2.5812970995903015 0loss/SRqqt3/assignment_loss‚1.60194697201252 /loss/FBt3/detection_loss‚2.5217370646340505 % _timestamp‚1.7884448938031664e+09 0loss/FRt3/assignment_loss‚3.0204401329944006 /loss/FRt4/assignment_loss‚2.865476514611925 2loss/SRqqt2/assignment_loss‚1.3910012413230208 1loss/FBt2/assignment_loss‚0.12219505965994257 1loss/FBt3/assignment_loss‚0.09814379103564923 1loss/FBt4/assignment_loss‚0.23345425797378994 1loss/SRqqt2/detection_loss‚1.7971079837944772Èø_Ù\“ "‚(:ˆ ÞæÔªø¤É gpu.0.gpu‚51.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚212.445' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.489 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â+8%E¡"‚(:º ÞæÔÃÀ¤Ë network.recv‚ 2.659119e+06 disk.rbd1.out‚ 3.6796875 disk.sda.in‚0.5625 disk.sda.out‚ 265.67578125 disk.sda2.in‚0! disk.nvme0n1.in‚ 1755.46484375% disk./.usageGB‚1107.7535972595215 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.in‚0 disk.sda3.out‚ 265.67578125# disk.dm-0.out‚65178.36181640625 disk.dm-1.out‚ 265.67578125! disk.rbd2.out‚598.41259765625 disk.sda1.in‚0( disk./.usagePercent‚74.3422733792634# disk.nvme0c0n1.in‚ 1755.46484375) proc.memory.availableMB‚ 1871.58984375 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚65178.36181640625 network.sent‚ 2.1674691e+07 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚16.632914543151855 cpu‚25.335602405801787 proc.cpu.threads‚17 disk.rbd0.in‚0( disk.nvme0c0n1.out‚65178.36181640625 disk.sda2.out‚0$ memory_percent‚88.57672214508057# proc.memory.rssMB‚ 2725.13671875/ëfÙž¢"‚(:“ íæÔÎƳÉ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.448' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.48959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ɔ¢%'£"‚(:œ íæÔìÀ¨Ì& proc.memory.availableMB‚ 1871.59375( disk./.usagePercent‚74.3422733792634 disk.dm-1.out‚266.25 disk.sda.out‚266.25# proc.memory.rssMB‚ 2725.13671875 network.recv‚ 2.666231e+06 disk.sda1.out‚0 disk.dm-1.in‚0.5625* proc.memory.percent‚16.632914543151855 network.sent‚ 2.1784358e+07 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0! disk.rbd2.out‚599.98681640625 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.in‚0 disk.dm-0.out‚65179.23046875 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.6796875 proc.cpu.threads‚17% disk./.usageGB‚1107.7535972595215 disk.rbd0.out‚0% disk.nvme0c0n1.out‚65179.23046875 disk.sda3.out‚266.25 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.dm-0.in‚ 1755.46484375 disk.rbd2.in‚0 cpu‚25.334980396356652 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚65179.23046875# disk.nvme0c0n1.in‚ 1755.46484375$ memory_percent‚88.57669830322266¸õFÞ“¤"‚(:ˆ üæÔꇠÉ gpu.0.gpu‚37.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚91.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñO¡B¥"‚(:· üæÔ¯Ï Ë network.recv‚ 2.684368e+06' disk.nvme0c0n1.out‚65179.6845703125 disk.dm-1.in‚0.5625 disk.sda1.in‚0$ memory_percent‚88.57407569885254% disk./.usageGB‚1107.7535438537598 disk.sda.in‚0.5625 disk.sda1.out‚0" disk.dm-0.out‚65179.6845703125! disk.rbd2.out‚600.14404296875# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.out‚ 3.6796875# proc.memory.rssMB‚ 2725.13671875 cpu‚25.334409638208122) disk./.usagePercent‚74.34226979515792 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1755.46484375* proc.memory.percent‚16.632914543151855 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.sda3.out‚ 266.67578125 disk.sda2.in‚0% disk.nvme0n1.out‚65179.6845703125 network.sent‚ 2.1797216e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46484375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 266.67578125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 266.67578125( proc.memory.availableMB‚ 1872.0234375ÊÜqO2¦"- _wandbruntime‚7372  _runtime‚7372dy:ŧ"¿ % lr-AdamW/pg1‚8.56786259981704e-05 % lr-AdamW/pg2‚8.56786259981704e-05 trainer/global_step‚8699 % _timestamp‚1.7884449327151318e+09 _runtime‚7372.759805747 _step‚713½ÿóiʨ"Ä _step‚713 % lr-AdamW/pg1‚8.56786259981704e-05 % lr-AdamW/pg2‚8.56786259981704e-05 trainer/global_step‚8699 % _timestamp‚1.7884449327151318e+09 _runtime‚7372.759805747É/püô2©"- _wandbruntime‚7372  _runtime‚7372n0(ª"ŠP YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚972199671767006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16124687504994 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚987789981267456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128665011 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971638502641664 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚972197048054370 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚971638502641664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16154078835200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚831165339548672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚956072385977344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16127285789662 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _step‚714 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987789981267456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚831165311414272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162428474 % _timestamp‚1.7884449327158794e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128665005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143908786 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚853281993708032 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚971638529033216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚484 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093485 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143908302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83504669 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128665011 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83504666 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22116732884480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚987792607868416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚853282072433152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128665005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _runtime‚7372.760751349 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16151478625792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839ö* q(«"P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚971638529033216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚484 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291093485 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143908302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83504669 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128665011 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83504666 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22116732884480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚987792607868416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚853282072433152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128665005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _runtime‚7372.760751349 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16151478625792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚972199671767006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16124687504994 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚987789981267456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128665011 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971638502641664 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚972197048054370 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚971638502641664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16154078835200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚831165339548672 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚956072385977344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16127285789662 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _step‚714 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987789981267456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚831165311414272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162428474 % _timestamp‚1.7884449327158794e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128665005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143908786 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚853281993708032Êcâ792¬"- _wandbruntime‚7373  _runtime‚7373¢«Íï­"”N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128679247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/rŠ9®+%equested_bytes.small_pool.allocated‚16129115132538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚853376888433664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22119188965888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚971746468738560 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128679241 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚853376967145472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60410695 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 trainer/global_step‚8699 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16155911177216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16153310954496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162446734 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚971746468738560 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143924093 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚956178727151616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚715 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16153310954496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚987902379915776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7373.501298617 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128679247 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚987902379915776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 % _timestamp‚1.7884449334561913e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚831257727602176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125981 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128679241 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143924560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83513865 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚971746442347008 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚956178701723648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16155911177216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60410231 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291105915 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚971746442347008†Ï†¤'®"™N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚956178701723648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16155911177216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60410231 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291105915 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚971746442347008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128679247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16129115132538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚853376888433664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22119188965888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚971746468738560 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128679241 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚853376967145472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60410695 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 trainer/global_step‚8699 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16155911177216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16153310954496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162446734 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚971746468738560 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143924093 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚956178727151616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚715 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16153310954496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚987902379915776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚7373.501298617 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128679247 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚987902379915776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 % _timestamp‚1.7884449334561913e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚831257727602176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125981 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128679241 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143924560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83513865 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚971746442347008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚977035487908864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚977035514300416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚977605587492578 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961389419262976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚977035487908864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292720337 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163343526 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚993281223984640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚718 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16216168229602 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144706609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16243109474816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚977608211205214 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993278597383680 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22241928152576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16218766514270 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034Õ·Äã(½"P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16218766514270 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163323492 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚977035487908864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚977035514300416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚977605587492578 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961389419262976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚977035487908864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292720337 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163343526 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚993281223984640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚718 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16216168229602 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144706609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16243109474816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚977608211205214 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993278597383680 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22241928152576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672Îü츤2¾"- _wandbruntime‚7413  _runtime‚7413°ú'¿"“N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129391041 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129391047 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚835877102142976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83973662 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129391047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163361786 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163341726 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961495785865216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16247542026240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16244941803520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163341726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22244445832704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993390996032000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16247542026240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚858121624590336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _runtime‚7413.372118466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚977713758009518 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144722100 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚977716381722362 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16220595857146 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚977143454005760 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚977143454005760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16217997572270 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144722540 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚977143427614208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 $ _timestamp‚1.788444973326988e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292732767 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163361786 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚835877130277376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22244494312960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292732767 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129391041 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚961495760437248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚993388369417728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124îpŠÈFÀ"˜N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83973662 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129391047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163361786 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163341726 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961495785865216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16247542026240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16244941803520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163341726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22244445832704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993390996032000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16247542026240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚858121624590336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _runtime‚7413.372118466 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚977713758009518 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144722100 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚977716381722362 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16220595857146 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚977143454005760 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚977143454005760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16217997572270 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144722540 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚977143427614208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 $ _timestamp‚1.788444973326988e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292732767 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163361786 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚835877130277376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22244494312960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292732767 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129391041 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚961495760437248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚993388369417728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129391041 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 KEDeviceSÊðØ tatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129391047 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚835877102142976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920ÏèYèb2Á"- _wandbruntime‚7413  _runtime‚7413kfŸ£[Â"Õ 2loss/SRqqt2/assignment_loss‚1.2337714682139878 1loss/SRqqt2/detection_loss‚1.7541538793242202 /loss/FBt1/detection_loss‚0.6916436354275158 0loss/SRqqt4/assignment_loss‚1.62239304313866 0loss/FBt4/assignment_loss‚0.1015167370904237 1loss/SRqqt4/detection_loss‚1.7744332486047194 2loss/SRqqt3/assignment_loss‚1.7173101063340137 /loss/FRt1/detection_loss‚2.0197231886518554 /loss/FRt3/detection_loss‚2.5743236531351936 1loss/SRqqt1/detection_loss‚1.1496513713347285 _runtime‚7413.373123329 0loss/FRt4/assignment_loss‚2.9929230511188507 2loss/SRqqt1/assignment_loss‚0.7291837587233698 /loss/FRt4/detection_loss‚2.2016709111630917 epoch‚7 _step‚720 0loss/FRt1/assignment_loss‚1.4157403430273368 /loss/FRt2/detection_loss‚2.2916280028033764 .loss/FBt2/detection_loss‚2.710352376103401 %loss/total_loss‚42.42315699311952 /loss/FRt3/assignment_loss‚2.917855706907088 1loss/FBt1/assignment_loss‚0.14324353260033604 0loss/SRqqt3/detection_loss‚2.080920013097616 -loss/FBt3/detection_loss‚4.89743306239446 trainer/global_step‚8749 1loss/FBt3/assignment_loss‚0.21367448723564544 % _timestamp‚1.7884449733285878e+09 0loss/FRt2/assignment_loss‚2.4060053280059326 1loss/FBt2/assignment_loss‚0.14721505694160442 /loss/FBt4/detection_loss‚2.63639103174209582Ln`Ã"Ú 0loss/FRt2/assignment_loss‚2.4060053280059326 1loss/FBt2/assignment_loss‚0.14721505694160442 /loss/FBt4/detection_loss‚2.6363910317420958 2loss/SRqqt2/assignment_loss‚1.2337714682139878 1loss/SRqqt2/detection_loss‚1.7541538793242202 /loss/FBt1/detection_loss‚0.6916436354275158 0loss/SRqqt4/assignment_loss‚1.62239304313866 0loss/FBt4/assignment_loss‚0.1015167370904237 1loss/SRqqt4/detection_loss‚1.7744332486047194 2loss/SRqqt3/assignment_loss‚1.7173101063340137 /loss/FRt1/detection_loss‚2.0197231886518554 /loss/FRt3/detection_loss‚2.5743236531351936 1loss/SRqqt1/detection_loss‚1.1496513713347285 _runtime‚7413.373123329 0loss/FRt4/assignment_loss‚2.9929230511188507 2loss/SRqqt1/assignment_loss‚0.7291837587233698 /loss/FRt4/detection_loss‚2.2016709111630917 epoch‚7 _step‚720 0loss/FRt1/assignment_loss‚1.4157403430273368 /loss/FRt2/detection_loss‚2.2916280028033764 .loss/FBt2/detection_loss‚2.710352376103401 %loss/total_loss‚42.42315699311952 /loss/FRt3/assignment_loss‚2.917855706907088 1loss/FBt1/assignment_loss‚0.14324353260033604 0loss/SRqqt3/detection_loss‚2.080920013097616 -loss/FBt3/detection_loss‚4.89743306239446 trainer/global_step‚8749 1loss/FBt3/assignment_loss‚0.21367448723564544 % _timestamp‚1.7884449733285878e+09ÐÓÌd”Ä"‚(:‰ ¸‚æÔß•„É gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚229.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óšð<Å"‚(:± ¸‚æÔ‰ðÁË disk.dm-1.in‚0.5625 disk.sda1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚65383.87548828125 proc.cpu.threads‚17 network.recv‚ 2.73847e+06 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 268.6640625 disk.rbd3.out‚0# disk.dm-0.out‚65383.87548828125 disk.drbd1777.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2725.13671875 disk.dm-0.in‚ 1755.46484375) disk./.usagePercent‚74.35145329742619 disk.rbd1.in‚0 disk.rbd2.out‚ 605.0859375& disk.nvme0n1.out‚65383.87548828125 disk.rbd0.in‚0 network.sent‚ 2.1941072e+07 disk.rbd1.out‚ 3.6796875 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 268.6640625* proc.memory.percent‚16.632914543151855% disk./.usageGB‚1107.8903846740723 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1755.46484375# memory_percent‚88.5775089263916 disk.sda.out‚ 268.6640625 disk.sda1.out‚0! disk.nvme0n1.in‚ 1755.46484375( proc.memory.availableMB‚ 1871.4609375 cpu‚25.332151398074988ƽ'ŸÆ"‚(:” Ç‚æÔÎæœÉ gpu.0.gpu‚81.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚260.787' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.157399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è8i€@Ç"‚(:µ Ç‚æÔ‚ÊýÊ( disk.nvme0c0n1.out‚65481.93701171875 disk.sda.in‚0.5625& disk.nvme0n1.out‚65481.93701171875# disk.nvme0c0n1.in‚ 1755.46484375 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 network.sent‚ 2.2050551e+07) disk./.usagePercent‚74.35146097765222 disk.dm-1.out‚ 269.53515625$ memory_percent‚88.57920169830322 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 cpu‚25.33158837795201 proc.cpu.threads‚17% disk./.usageGB‚1107.8904991149902 disk.sda3.out‚ 269.53515625 disk.drbd1777.out‚0 disk.rbd2.out‚ 605.109375 disk.sda1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚65481.93701171875 disk.rbd1.out‚ 3.6796875 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1755.46484375 disk.sda.out‚ 269.53515625* proc.memory.percent‚16.632914543151855 network.recv‚ 2.745106e+06! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.out‚0 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 1871.18359375# proc.memory.rssMB‚ 2725.13671875ŠÑ¢(2È"- _wandbruntime‚7452  _runtime‚7452™ëdÅÉ"¿ % lr-AdamW/pg1‚8.10812752563542e-05 % lr-AdamW/pg2‚8.10812752563542e-05 trainer/global_step‚8799 % _timestamp‚1.7884450119977658e+09 _runtime‚7452.042422939 _step‚721w0ÇÊÊ"Ä % lr-AdamW/pg1‚8.10812752563542e-05 % lr-AdamW/pg2‚8.10812752563542e-05 trainer/global_step‚8799 % _timestamp‚1.7884450119977658e+09 _runtime‚7452.042422939 _step‚721Ñþ¦;b2Ë"- _wandbruntime‚7452  _runtime‚7452û÷ÎÕ(Ì"ŠP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚982432499567616 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚840404144899072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16310247238878 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164258578 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚982432473176064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚982432499567616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61079771 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61079275 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16334740323840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294347189 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130088611 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚983016750643422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚983014126930786 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164238544 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚998769840100864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145504040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚966706477976576 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚862771721822720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚862771641000448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16307648954210 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16337340533248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚966706503404544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚998767213499904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22367524235776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚998767213499904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16334740323840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130088611 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚982432473176064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84424266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22367576923648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649±-4(Í"P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚982432473176064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84424266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22367576923648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚982432499567616 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚840404144899072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16310247238878 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164258578 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚982432473176064 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚982432499567616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61079771 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61079275 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16334740323840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294347189 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130088611 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚983016750643422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚983014126930786 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164238544 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚998769840100864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145504040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚966706477976576 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚862771721822720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚862771641000448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16307648954210 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16337340533248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚966706503404544 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚998767213499904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22367524235776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚998767213499904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16334740323840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130088611 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074Òb?½2Î"- _wandbruntime‚7452  _runtime‚7452+a-Ò Ï"“N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚982540439272960 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145519414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164256778 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22370069424640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130102841 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16336572652544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚982540439272960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130102847 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84433462 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22370018847232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚966812819150848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164256778 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚7452.776975527 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚998879612148224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61086421 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?gÏßG5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84433465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130102841 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚862866602377216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379685 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _step‚723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164276838 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16312076581754 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 $ _timestamp‚1.788445012731715e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚983122297447726 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294359619 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚983124921160570 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚966812844578816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130102847 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164276838 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚982540412881408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16339172875264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61085952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚862866523665408 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16309478296878 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145519886au<0'Ð"˜N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61086421 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84433465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130102841 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚862866602377216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379685 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _step‚723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164276838 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16312076581754 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 $ _timestamp‚1.788445012731715e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚983122297447726 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294359619 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚983124921160570 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚966812844578816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130102847 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164276838 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚982540412881408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16339172875264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61085952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚862866523665408 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16309478296878 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145519886 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚982540439272960 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145519414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164256778 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22370069424640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130102841 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16336572652544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚982540439272960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130102847 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84433462 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22370018847232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚966812819150848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164256778 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚7452.776975527 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚998879612148224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568ÓúO¢ž2Ñ"- _wandbruntime‚7452  _runtime‚7452`à}]Ò"× /loss/FRt1/detection_loss‚1.3939491759908076 .loss/FRt2/detection_loss‚2.663475309809049 /loss/FRt4/detection_loss‚2.4265204599270453 1loss/SRqqt1/detection_loss‚0.9095304387737997 % _timestamp‚1.7884450127337055e+09 /loss/FRt3/assignment_loss‚2.602836854317609 3loss/FBt3/assignment_loss‚0.0014893979096086696 0loss/SRqqt4/detection_loss‚1.714985854266321 /loss/FBt2/detection_loss‚2.2794929802417756 0loss/FRt1/assignment_loss‚1.4101619807382424 2loss/SRqqt4/assignment_loss‚1.2202376543667497 0loss/FBt4/assignment_loss‚0.2007518059836002 /loss/FBt1/detection_loss‚0.4883624408778149 %loss/total_loss‚39.23419641146981 trainer/global_step‚8799 _step‚724 .loss/FBt4/detection_loss‚2.072517141699791 0loss/FRt2/assignment_loss‚3.0813934521542654 0loss/FRt4/assignment_loss‚2.0833842234733777 1loss/SRqqt3/assignment_loss‚1.325297593927154 1loss/SRqqt2/detection_loss‚1.8840845633784067 1loss/FBt1/assignment_loss‚0.14314887229751694 0loss/SRqqt3/detection_loss‚2.718522209387559 _runtime‚7452.778291267 /loss/FRt3/detection_loss‚2.3661341647624368 2loss/SRqqt2/assignment_loss‚1.2907298781058711 /loss/FBt3/detection_loss‚3.7661019191145897 epoch‚7 2loss/SRqqt1/assignment_loss‚0.9365144263283582 0loss/FBt2/assignment_loss‚0.2545736136380583,µbbÓ"Ü /loss/FRt3/assignment_loss‚2.602836854317609 3loss/FBt3/assignment_loss‚0.0014893979096086696 0loss/SRqqt4/detection_loss‚1.714985854266321 /loss/FBt2/detection_loss‚2.2794929802417756 0loss/FRt1/assignment_loss‚1.4101619807382424 2loss/SRqqt4/assignment_loss‚1.2202376543667497 0loss/FBt4/assignment_loss‚0.2007518059836002 /loss/FBt1/detection_loss‚0.4883624408778149 %loss/total_loss‚39.23419641146981 trainer/global_step‚8799 _step‚724 .loss/FBt4/detection_loss‚2.072517141699791 0loss/FRt2/assignment_loss‚3.0813934521542654 0loss/FRt4/assignment_loss‚2.0833842234733777 1loss/SRqqt3/assignment_loss‚1.325297593927154 1loss/SRqqt2/detection_loss‚1.8840845633784067 1loss/FBt1/assignment_loss‚0.14314887229751694 0loss/SRqqt3/detection_loss‚2.718522209387559 _runtime‚7452.778291267 /loss/FRt3/detection_loss‚2.3661341647624368 2loss/SRqqt2/assignment_loss‚1.2907298781058711 /loss/FBt3/detection_loss‚3.7661019191145897 epoch‚7 2loss/SRqqt1/assignment_loss‚0.9365144263283582 0loss/FBt2/assignment_loss‚0.2545736136380583 /loss/FRt1/detection_loss‚1.3939491759908076 .loss/FRt2/detection_loss‚2.663475309809049 /loss/FRt4/detection_loss‚2.4265204599270453 1loss/SRqqt1/detection_loss‚0.9095304387737997 % _timestamp‚1.7884450127337055e+09Ôg`÷Y”Ô"‚(:‰ Ö‚æÔ gpu.0.gpu‚70.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚222.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.4406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'43?Õ"‚(:´ Ö‚æÔ§« Ì disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚65482.24951171875 cpu‚25.33101488700256 network.recv‚ 2.748975e+06 disk.drbd1777.in‚0 disk.sda.out‚ 269.7578125 network.sent‚ 2.2055695e+07 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1755.46484375 disk.dm-1.out‚ 269.7578125 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚88.57762813568115# disk.dm-0.out‚65482.24951171875) disk./.usagePercent‚74.35146097765222 disk.drbd1777.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚65482.24951171875# proc.memory.rssMB‚ 2725.13671875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0.5625* proc.memory.percent‚16.632914543151855% disk./.usageGB‚1107.8904991149902 disk.rbd3.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375) proc.memory.availableMB‚ 1871.44140625 disk.rbd1.out‚ 3.6796875 disk.rbd2.out‚ 606.76171875 disk.sda.in‚0.5625 disk.sda3.out‚ 269.7578125 proc.cpu.threads‚17£úéý”Ö"‚(:‰ å‚æÔ­ÇÿÈ gpu.0.gpu‚68.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.208' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.4416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÿ9×"‚(:® å‚æÔ”ΙË% disk.nvme0n1.out‚65483.2880859375' disk.nvme0c0n1.out‚65483.2880859375 disk.dm-0.in‚ 1755.46484375' proc.memory.availableMB‚ 1871.640625 proc.cpu.threads‚17( disk./.usagePercent‚74.3683083214655 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 270.56640625 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.6796875 disk.rbd2.out‚ 606.7890625 disk.sda1.in‚0# proc.memory.rssMB‚ 2725.13671875% disk./.usageGB‚1108.1415367126465 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.rbd2.in‚0 cpu‚25.3304266876257 network.recv‚ 2.75542e+06 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1755.46484375 disk.sda3.out‚ 270.56640625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 network.sent‚ 2.2164692e+07" disk.dm-0.out‚65483.2880859375 disk.dm-1.out‚ 270.56640625$ memory_percent‚88.57641220092773* proc.memory.percent‚16.632914543151855 disk.drbd1777.in‚0 disk.rbd1.in‚0–¾ª”Ø"‚(:‰ ô‚æÔ¶ï¦É gpu.0.gpu‚59.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚203.857' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.7714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æiJ<Ù"‚(:± ô‚æÔ¥­¼Ë disk.sda3.out‚ 270.88671875 disk.dm-0.in‚ 1755.46484375 disk.sda.in‚0.5625" disk.dm-0.out‚65580.7802734375' proc.memory.availableMB‚ 1871.640625# proc.memory.rssMB‚ 2725.13671875 disk.rbd0.in‚0 disk.sda.out‚ 270.88671875 disk.sda1.out‚0! disk.nvme0n1.in‚ 1755.46484375 disk.dm-1.in‚0.5625( disk./.usagePercent‚74.3682304951751 disk.rbd1.out‚ 3.6796875* proc.memory.percent‚16.632914543151855 network.recv‚ 2.759541e+06 disk.rbd3.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚65580.7802734375$ memory_percent‚88.57641220092773 disk.rbd2.out‚ 608.4453125 disk.sda2.out‚0% disk.nvme0n1.out‚65580.7802734375 disk.sda3.in‚0.5625 disk.rbd0.out‚0 network.sent‚ 2.2170597e+07% disk./.usageGB‚1108.1403770446777 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 270.88671875 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 cpu‚25.329898072777123 proc.cpu.threads‚17Qbî2Ú"- _wandbruntime‚7492  _runtime‚7492@=¸2ÆÛ"À & lr-AdamW/pg1‚7.881758375666917e-05 & lr-AdamW/pg2‚7.881758375666917e-05 trainer/global_step‚8849 $ _timestamp‚1.788445052033211e+09 _runtime‚7492.077927867 _step‚725V‡ ØËÜ"Å _runtime‚7492.077927867 _step‚725 & lr-AdamW/pg1‚7.881758375666917e-05 & lr-AdamW/pg2‚7.881758375666917e-05 trainer/global_step‚8849 $ _timestamp‚1.788445052033211e+09ÕªÖô2Ý"- _wandbruntime‚7492  _runtime‚7492ÿÕÞ"ŽP =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚987829458443264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130800411 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61417746 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84884066 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16426371172864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845023547574272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1004255829616128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165173630 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16399129678818 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972023536690176 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 _runtime‚7492.078893386 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1004258456217088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165153596 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚987829484834816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚988425290081630 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚867516798774784 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1004258456217088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1004255829616128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130800405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466t_à¤(ß"“P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16401727963486 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146301812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130800405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚987829458443264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130800411 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61417746 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84884066 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16426371172864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845023547574272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1004255829616128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165173630 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16399129678818 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972023536690176 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 _runtime‚7492.078893386 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1004258456217088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165153596 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚987829484834816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚988425290081630 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚867516798774784 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1004258456217088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1004255829616128 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146317840 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚988530836885934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61425049 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146318314 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130814647 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚867611693924352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16403557306362 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130814641 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚867611772636160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16400959021486 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚987937424540160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16430803724288 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165171830 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61424578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1004368228264448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130814641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84893262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚845115907493376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 _runtime‚7492.820518847 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22495786430976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130814647 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚987937398148608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚987937424540160 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚988533460598778 trainer/global_step‚8849 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22495837008384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972129903292416 % _timestamp‚1.7884450527751977e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1004365601650176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚987937398148608 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16428203501568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16430803724288 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130814647 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚867611693924352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16403557306362 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130814641 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚867611772636160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16400959021486 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚987937424540160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16430803724288 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165171830 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61424578 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1004368228264448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130814641 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84893262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚845115907493376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 _runtime‚7492.820518847 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22495786430976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130814647 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚987937398148608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚987937424540160 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚988533460598778 trainer/global_step‚8849 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22495837008384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972129903292416 % _timestamp‚1.7884450527751977e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1004365601650176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚987937398148608 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16428203501568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16430803724288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146317840 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚988530836885934 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61425049 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146318314 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496ך7l²2ã"- _wandbruntime‚7492  _runtime‚7492 ³® Òä"Ú _step‚728 0loss/FRt1/assignment_loss‚0.8217606599684115 0loss/FRt4/assignment_loss‚2.1837772207382398 1loss/FBt2/assignment_loss‚0.14879653216876948 1loss/FBt4/assignment_loss‚0.2884­âGŽ7702077057746 1loss/SRqqt1/detection_loss‚0.8508527027026042 /loss/FRt3/detection_loss‚2.4305979445958252 0loss/SRqqt4/detection_loss‚1.859138522864036 %loss/total_loss‚37.56373445411148 epoch‚7 _runtime‚7492.821753773 2loss/SRqqt3/assignment_loss‚1.6445514027248411 1loss/FBt1/assignment_loss‚0.16830009808974694 .loss/FBt4/detection_loss‚2.182536917073386 2loss/SRqqt1/assignment_loss‚0.7309152900402907 2loss/SRqqt2/assignment_loss‚1.2711686829850077 2loss/SRqqt4/assignment_loss‚1.2210942023312268 /loss/FBt1/detection_loss‚0.5546508006446845 3loss/FBt3/assignment_loss‚0.0012413892953190952 .loss/FRt4/detection_loss‚2.428116112947464 1loss/SRqqt2/detection_loss‚1.7029621896147729 trainer/global_step‚8849 /loss/FBt3/detection_loss‚3.0320222854614256 % _timestamp‚1.7884450527771664e+09 0loss/FRt2/assignment_loss‚2.4730374881854424 0loss/FRt3/assignment_loss‚3.4746083633767233 .loss/FRt1/detection_loss‚1.289071760174853 /loss/FRt2/detection_loss‚2.1231343069901834 0loss/SRqqt3/detection_loss‚2.376342635820894 /loss/FBt2/detection_loss‚2.3065799245467553((VÅeå"ß 2loss/SRqqt2/assignment_loss‚1.2711686829850077 2loss/SRqqt4/assignment_loss‚1.2210942023312268 /loss/FBt1/detection_loss‚0.5546508006446845 3loss/FBt3/assignment_loss‚0.0012413892953190952 .loss/FRt4/detection_loss‚2.428116112947464 1loss/SRqqt2/detection_loss‚1.7029621896147729 trainer/global_step‚8849 /loss/FBt3/detection_loss‚3.0320222854614256 % _timestamp‚1.7884450527771664e+09 0loss/FRt2/assignment_loss‚2.4730374881854424 0loss/FRt3/assignment_loss‚3.4746083633767233 .loss/FRt1/detection_loss‚1.289071760174853 /loss/FRt2/detection_loss‚2.1231343069901834 0loss/SRqqt3/detection_loss‚2.376342635820894 /loss/FBt2/detection_loss‚2.3065799245467553 _step‚728 0loss/FRt1/assignment_loss‚0.8217606599684115 0loss/FRt4/assignment_loss‚2.1837772207382398 1loss/FBt2/assignment_loss‚0.14879653216876948 1loss/FBt4/assignment_loss‚0.28847702077057746 1loss/SRqqt1/detection_loss‚0.8508527027026042 /loss/FRt3/detection_loss‚2.4305979445958252 0loss/SRqqt4/detection_loss‚1.859138522864036 %loss/total_loss‚37.56373445411148 epoch‚7 _runtime‚7492.821753773 2loss/SRqqt3/assignment_loss‚1.6445514027248411 1loss/FBt1/assignment_loss‚0.16830009808974694 .loss/FBt4/detection_loss‚2.182536917073386 2loss/SRqqt1/assignment_loss‚0.7309152900402907Ø%Z×ß”æ"‚(:‰ ƒƒæÔø¼¡É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚216.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.3056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–,Iú3ç"‚(:¨ ƒƒæÔ›ì÷Ë disk.sda1.out‚0* proc.memory.percent‚16.632914543151855 disk.dm-0.in‚ 1755.46484375 disk.dm-0.out‚65582.16015625% disk./.usageGB‚1108.1405792236328 disk.drbd1777.in‚0) proc.memory.availableMB‚ 1871.60546875! disk.nvme0n1.in‚ 1755.46484375 disk.sda3.in‚0.5625 disk.rbd1.in‚0 disk.sda2.out‚0 network.sent‚ 2.2176827e+07 disk.rbd2.in‚0$ memory_percent‚88.57662677764893# proc.memory.rssMB‚ 2725.13671875 cpu‚25.32930397372236 proc.cpu.threads‚17 disk.sda3.out‚ 271.6015625 disk.rbd2.out‚ 608.46875 disk.sda.in‚0.5625 disk.rbd3.in‚0 network.recv‚ 2.763976e+06% disk.nvme0c0n1.out‚65582.16015625 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 271.6015625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0) disk./.usagePercent‚74.36824406357442# disk.nvme0n1.out‚65582.16015625 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-1.out‚ 271.6015625 disk.drbd1777.out‚0 disk.rbd0.in‚0^ ãŸè"‚(:” ’ƒæÔÒ³ªÉ gpu.0.gpu‚62.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚131.907' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.381400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ„/âCé"‚(:¸ ’ƒæÔúüšÌ disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚65679.68408203125$ memory_percent‚88.57815265655518 network.sent‚ 2.2285225e+07! disk.rbd2.out‚610.13720703125 disk.sda2.out‚0 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 1871.35546875 cpu‚25.328746450642544 disk.rbd1.out‚ 3.6796875 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚65679.68408203125 disk.dm-0.in‚ 1755.46484375* proc.memory.percent‚16.632914543151855 disk.rbd3.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2725.13671875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1755.46484375 network.recv‚ 2.769941e+06 disk.sda.out‚ 271.9609375 disk.sda1.out‚0 disk.sda3.out‚ 271.9609375 proc.cpu.threads‚17 disk.dm-1.out‚ 271.9609375% disk./.usageGB‚1108.1406364440918! disk.nvme0n1.in‚ 1755.46484375) disk./.usagePercent‚74.36824790368742 disk.rbd1.in‚0& disk.nvme0n1.out‚65679.68408203125 disk.rbd0.in‚0 disk.drbd1777.in‚0Ò0¨”ê"‚(:‰ ¡ƒæÔå­ûÈ gpu.0.gpu‚78.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚246.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.3546 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ÍÓf'ë"‚(:œ ¡ƒæÔäôãË disk.dm-0.out‚ 65680.703125 disk.rbd2.in‚0! disk.rbd2.out‚610.15673828125 disk.sda.in‚0.5625! disk.nvme0n1.out‚ 65680.703125 disk.sda3.out‚272.375 disk.rbd0.out‚0% disk./.usageGB‚1108.1405792236328 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1755.46484375# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.out‚ 3.6796875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46484375$ memory_percent‚88.57667446136475 network.recv‚ 2.774238e+06 disk.sda1.in‚0* proc.memory.percent‚16.632914543151855) disk./.usagePercent‚74.36824406357442 disk.rbd1.in‚0 cpu‚25.32816303961262 network.sent‚ 2.2291214e+07 disk.drbd1777.out‚0 disk.sda3.in‚0.5625# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17 disk.sda.out‚272.375) proc.memory.availableMB‚ 1871.59765625 disk.sda2.out‚0# disk.nvme0c0n1.out‚ 65680.703125 disk.dm-1.in‚0.5625 disk.dm-1.out‚272.375 uÒ$2ì"- _wandbruntime‚7532  _runtime‚7532ÝÐÇí"Á _runtime‚7532.734073427 _step‚729 & lr-AdamW/pg1‚7.657784546885425e-05 & lr-AdamW/pg2‚7.657784546885425e-05 trainer/global_step‚8899 % _timestamp‚1.7884450926893785e+09Qç«Ìî"Æ % _timestamp‚1.7884450926893785e+09 _runtime‚7532.734073427 _step‚729 & lr-AdamW/pg1‚7.657784546885425e-05 & lr-AdamW/pg2‚7.657784546885425e-05 trainer/global_step‚8899Ù}Kn2ï"- _wandbruntime‚7532  _runtime‚7532ówÀF(ð"P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166088682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993226443710464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993226470102016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131512205 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61755821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22618863823360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16518002021888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147099690 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1009747072333312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚993226470102016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131512205 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166088682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61755353 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85343869 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580853 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1009744445732352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131512211 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16493208688094 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147099219 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009744445732352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166068648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009747072333312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚977340620831744 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16520602231296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85343866 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672<*œÐ(ñ"’P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166088682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚993226443710464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚993226470102016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131512205 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61755821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22618863823360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16518002021888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147099690 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1009747072333312 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚993226470102016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131512205 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166088682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61755353 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85343869 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580853 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1009744445732352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131512211 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16493208688094 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147099219 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009744445732352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166068648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009747072333312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚977340620831744 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16520602231296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85343866 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672ÚR!Øš2ò"- _wandbruntime‚7533  _runtime‚7533 áåó"—N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚872356639911936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22621329743360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1009856844380672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16522434573312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚993942000036986 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131526441 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚993939376324142 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 $ _timestamp‚1.788445093455686e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166086882 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.al1"8located‚ 166106942 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22621378223616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009856844380672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993334383415808 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚731 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993334383415808 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16519834350592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166086882 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147115150 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚849735338302976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚872356716526592îÊ"'ô"œN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131526447 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61762088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131526441 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚7533.501041566 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1009854217766400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166106942 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22621378223616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009856844380672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚993334383415808 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚731 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚993334383415808 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16519834350592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166086882 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147115150 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚849735338302976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚872356716526592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚872356639911936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22621329743360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1009856844380672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16522434573312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚993942000036986 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131526441 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚993939376324142 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 $ _timestamp‚1.788445093455686e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166086882 Fû"‚(:³ ¿ƒæÔ·€¯Ì) disk./.usagePercent‚74.36824534361209# disk.nvme0c0n1.in‚ 1755.46484375& disk.nvme0n1.out‚65778.58349609375 cpu‚25.3270261020385 disk.sda3.in‚0.5625 disk.rbd1.in‚0 network.recv‚ 2.785444e+06 disk.sda3.out‚ 273.5703125 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1755.46484375* proc.memory.percent‚16.632914543151855( disk.nvme0c0n1.out‚65778.58349609375# disk.dm-0.out‚65778.58349609375 disk.dm-1.out‚ 273.5703125 disk.sda1.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 1871.3515625# proc.memory.rssMB‚ 2725.13671875 disk.sda.out‚ 273.5703125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.rbd2.out‚610.2587890625 proc.cpu.threads‚17% disk./.usageGB‚1108.1405982971191 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda.in‚0.5625 network.sent‚ 2.240622e+07 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚88.57817649841309c˜Ø2ü"- _wandbruntime‚7573  _runtime‚7573Î’÷>Äý"¾ $ _timestamp‚1.788445133483628e+09 _runtime‚7573.528324922 _step‚733 % lr-AdamW/pg1‚7.43625271510115e-05 % lr-AdamW/pg2‚7.43625271510115e-05 trainer/global_step‚8949åBÈÉþ"à _runtime‚7573.528324922 _step‚733 % lr-AdamW/pg1‚7.43625271510115e-05 % lr-AdamW/pg2‚7.43625271510115e-05 trainer/global_step‚8949 $ _timestamp‚1.788445133483628e+09Ýo’2ÿ"- _wandbruntime‚7573  _runtime‚7573+òW(€#ŽP \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16612233080320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167003734 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16582091128034 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147897199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚982657654117376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1015235688449536 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1015233061848576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 trainer/global_step‚8949 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚854262352924672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _runtime‚7573.529377193 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1015233061848576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62093530 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132224011 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚998623455369216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85803669 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62093063 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚982657679545344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85803666 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1015235688449536 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147897199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚982657654117376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1015235688449536 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1015233061848576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 trainer/global_step‚8949 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚854262352924672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _runtime‚7573.529377193 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1015233061848576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62093530Þð&Ó2‚#- _wandbruntime‚7574  _runtime‚7574H¬¾”'ƒ#˜N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85812862 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚854354740978176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167001934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62100143 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚999347915762350 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132238241 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16614065422336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚998731395074560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147912562 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884451342482703e+09 _step‚735 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132238241 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22746990704128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1015342833882624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167021994 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22747041281536 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚877101703547904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808lL-¡#'„#N PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147913008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚998731395074560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚999350539475194 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167021994 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22747041281536 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚877101703547904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85812862 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚854354740978176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167001934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62100143 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚999347915762350 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132238241 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16614065422336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚998731395074560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147912562 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884451342482703e+09 _step‚735 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132238241 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22746990704128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1015342833882624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1004020440636416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚881751782569472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132935811 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1004020440636416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86263466 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚987974738258944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132935805 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1004650908396254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86263469 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148692668 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167898752 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020721677964800 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1004020414244864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020724304565760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167918786 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1020724304565760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1004020440636416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚881751782569472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132935811 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1004020440636416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86263466 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚987974738258944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132935805 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1004650908396254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86263469 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148692668 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167898752 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020721677964800 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1004020414244864 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020724304565760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167918786 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1020724304565760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚858974143653376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16705696271360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148708469 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1004128353950208 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867027 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167916986 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚881846669744128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22872526090752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132950047 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1004128380341760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1004756455200558 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚988081079433216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I@µíÒ{ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚8999 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚739 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167916986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86272662 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1020834076613120 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1004128353950208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1004128380341760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16703096048640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1020831449998848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167937046 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚7615.109360665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300867027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚881846669744128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22872526090752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132950047 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1004128380341760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1004756455200558 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚988081079433216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚8999 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _step‚739 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167916986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86272662 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1020834076613120 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1004128353950208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1004128380341760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16703096048640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1020831449998848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167937046 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚7615.109360665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚858974143653376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16705696271360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148708469 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1004128353950208 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300867027 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167916986 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 E}ÒŸœ#‚(:” Š„æÔúÕøÉ gpu.0.gpu‚90.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚235.995' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.199000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡„û{C#‚(:¸ Š„æÔ†î½Ë disk.sda2.out‚0 disk.rbd3.in‚0% disk./.usageGB‚1108.6408882141113# disk.dm-0.out‚65975.08740234375 disk.dm-1.out‚ 276.34765625 disk.rbd1.out‚ 3.6796875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚ 2.809685e+06( disk.nvme0c0n1.out‚65975.08740234375 disk.sda2.in‚0 disk.sda3.out‚ 276.34765625 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.out‚0 disk.sda.in‚0.5625# proc.memory.rssMB‚ 2725.13671875 network.sent‚ 2.2640252e+07 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 276.34765625! disk.nvme0n1.in‚ 1755.46484375 disk.sda3.in‚0.5625 disk.rbd2.out‚ 612.169921875) disk./.usagePercent‚74.40182021971263# disk.nvme0c0n1.in‚ 1755.46484375$ memory_percent‚88.58132362365723 proc.cpu.threads‚17& disk.nvme0n1.out‚65975.08740234375 disk.rbd0.in‚0( proc.memory.availableMB‚ 1870.8359375* proc.memory.percent‚16.632914543151855 cpu‚25.324353074872796T•¾†“ž#‚(:ˆ ™„æÔ”—É gpu.0.gpu‚65.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚233.519' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.7038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02't±AŸ#‚(:¶ ™„æÔ¤¾™Ë disk.dm-1.out‚ 277.0078125 disk.rbd1.in‚0 disk.rbd2.out‚ 612.185546875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2725.13671875* proc.memory.percent‚16.632914543151855 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.6796875 disk.sda3.out‚ 277.0078125) disk./.usagePercent‚74.40182917997633 disk.dm-0.in‚ 1755.46484375 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚ 2.814235e+06! disk.nvme0n1.in‚ 1755.46484375& disk.nvme0n1.out‚65975.68212890625( disk.nvme0c0n1.out‚65975.68212890625$ memory_percent‚88.58144283294678 disk.dm-1.in‚0.5625 disk.sda1.in‚0 network.sent‚ 2.2646193e+07 disk.sda.out‚ 277.0078125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1870.81640625 cpu‚25.323786401271345 proc.cpu.threads‚17% disk./.usageGB‚1108.6410217285156# disk.dm-0.out‚65975.68212890625 disk.sda3.in‚0.5625Ê°2 #- _wandbruntime‚7654  _runtime‚7654Êö»Æ¡#À & lr-AdamW/pg2‚7.000699191601462e-05 trainer/global_step‚9049 $ _timestamp‚1.788445214944196e+09 _runtime‚7654.988850596 _step‚741 & lr-AdamW/pg1‚7.000699191601462e-05ä¤SË¢#Å & lr-AdamW/pg1‚7.000699191601462e-05 & lr-AdamW/pg2‚7.000699191601462e-05 trainer/global_step‚9049 $ _timestamp‚1.788445214944196e+09 _runtime‚7654.988850596 _step‚741å¾Ç›[2£#- _wandbruntime‚7654  _runtime‚76542z„5(¤#”P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62766984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚886496690815488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168813804 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302481449 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149490253 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚886496614187520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1026212920681984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1026210294081024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1009417399512064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1009417425903616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133647605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168833838 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚993291771544576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16767650861918 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚863501158275072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62766514 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1009417399512064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16795494778368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168813804 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86723266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22995532540416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚993291796972544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133647605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133647611 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 % _timestamp‚1.7884452149449615e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461409 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22995484046848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168833838 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62766514 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1009417399512064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16795494778368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168813804 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86723266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22995532540416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚993291796972544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133647605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133647611 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚†€½žÄ  2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 % _timestamp‚1.7884452149449615e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461409 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22995484046848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168833838 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1026210294081024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1009417399512064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1009417425903616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133647605 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168833838 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚993291771544576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16767650861918 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575æòUP–2¦#- _wandbruntime‚7655  _runtime‚7655g›<$'§#žN F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22998027618304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚886591497332224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16794726897664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1026322692729344 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1009525365608960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1026320066115072 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133661847 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86732465 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚993398138146816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16769480204794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚743 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168852098 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16766881919918 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149506146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168852098 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1009525339217408 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149505685 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚863593546328576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚863593518194176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133661841 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16797327120384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168832038 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1010167618351610 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚886591573946880 % _timestamp‚1.7884452157116234e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1009525339217408 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552yŠ&*)'¨#£N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133661841 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16797327120384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168832038 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1010167618351610 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚886591573946880 % _timestamp‚1.7884452157116234e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1009525339217408 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22998027618304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚886591497332224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16794726897664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1026322692729344 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1009525365608960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1026320066115072 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133661847 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86732465 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚993398138146816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16769480204794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚743 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168852098 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16766881919918 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149506146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168852098 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1009525339217408 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149505685 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚863593546328576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚863593518194176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920çùÝ-2©#- _wandbruntime‚7655  _runtime‚7655Tîp™Tª#Î 1loss/SRqqt2/assignment_loss‚1.202967123527612 .loss/FRt4/detection_loss‚2.640180031316621 0loss/FRt1/assignment_loss‚1.2794936464892492 0loss/FRt3/assignment_loss‚3.2262718203740244 0loss/FBt2/assignment_loss‚0.8212830864026728 0loss/FBt4/assignment_loss‚0.1859775169452405 /loss/FBt1/detection_loss‚0.6320666119691768 .loss/FBt2/detection_loss‚2.152566468715668 /loss/FRt4/assignment_loss‚2.682994098322732 2loss/SRqqt4/assignment_loss‚1.4576113107236657 2loss/SRqqt1/assignment_loss‚0.8425559702099882 /loss/FRt3/detection_loss‚2.5690434237722704 .loss/FBt4/detection_loss‚2.551255840063095 %loss/total_loss‚40.19431585436955 trainer/global_step‚9049 _step‚744 /loss/FBt1/assignment_loss‚0.138607249619938 0loss/FBt3/assignment_loss‚0.2840580369385342 .loss/FBt3/detection_loss‚3.001215491976057 /loss/FRt1/detection_loss‚1.7412203536886308 0loss/SRqqt2/detection_loss‚1.887609688121648 0loss/SRqqt3/detection_loss‚2.441121828224924 1loss/SRqqt3/assignment_loss‚1.266100207013854 0loss/SRqqt1/detection_loss‚0.716423270414057 1loss/SRqqt4/detection_loss‚1.9156133004942455 epoch‚7 0loss/FRt2/assignment_loss‚2.1188570288724677 /loss/FRt2/detection_loss‚2.4392224501731783 % _timestamp‚1.7884452157136233e+09 _runtime‚7655.758228668ñÙU6Y«#Ó 1loss/SRqqt2/assignment_loss‚1.202967123527612 .loss/FRt4/detection_loss‚2.640180031316621 0loss/FRt1/assignment_loss‚1.2794936464892492 0loss/FRt3/assignment_loss‚3.2262718203740244 0loss/FBt2/assignment_loss‚0.8212830864026728 0loss/FBt4/assignment_loss‚0.1859775169452405 /loss/FBt1/detection_loss‚0.6320666119691768 .loss/FBt2/detection_loss‚2.152566468715668 /loss/FRt4/assignment_loss‚2.682994098322732 2loss/SRqqt4/assignment_loss‚1.4576113107236657 2loss/SRqqt1/assignment_loss‚0.8425559702099882 /loss/FRt3/detection_loss‚2.5690434237722704 .loss/FBt4/detection_loss‚2.551255840063095 %loss/total_loss‚40.19431585436955 trainer/global_step‚9049 _step‚744 /loss/FBt1/assignment_loss‚0.138607249619938 0loss/FBt3/assignment_loss‚0.2840580369385342 .loss/FBt3/detection_loss‚3.001215491976057 /loss/FRt1/detection_loss‚1.7412203536886308 0loss/SRqqt2/detection_loss‚1.887609688121648 0loss/SRqqt3/detection_loss‚2.441121828224924 1loss/SRqqt3/assignment_loss‚1.266100207013854 0loss/SRqqt1/detection_loss‚0.716423270414057 1loss/SRqqt4/detection_loss‚1.9156133004942455 epoch‚7 0loss/FRt2/assignment_loss‚2.1188570288724677 /loss/FRt2/detection_loss‚2.4392224501731783 % _timestamp‚1.7884452157136233e+09 _runtime‚7655.758228668èÒüœŸ¬#‚(:” ¨„æÔžþ¨É gpu.0.gpu‚49.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.199' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.839800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ô6­#‚(:¶ ¨„æÔ©€ŽË( disk.nvme0c0n1.out‚66073.31787109375 disk.dm-1.out‚ 277.4453125$ memory_percent‚88.58034610748291) proc.memory.availableMB‚ 1870.99609375# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.out‚0 cpu‚25.323253081365053 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚66073.31787109375 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 277.4453125* proc.memory.percent‚16.632914543151855 network.sent‚ 2.2652033e+07% disk./.usageGB‚1108.6409530639648 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 277.4453125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚66073.31787109375 disk.drbd1777.in‚0) disk./.usagePercent‚74.40182457184072 disk.sda1.in‚0 network.recv‚ 2.818512e+06 Ðã³HA disk.rbd2.out‚ 613.884765625! disk.nvme0n1.in‚ 1755.46484375’Kt3”®#‚(:‰ ·„æÔ­Ô©É gpu.0.gpu‚86.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚225.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.1978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰‰/š,¯#‚(:¡ ·„æÔРƒË disk.rbd1.out‚ 3.6796875 disk.sda1.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 1870.484375 proc.cpu.threads‚17 disk.dm-1.out‚278.3125 disk.rbd3.in‚0# disk.nvme0n1.out‚66073.98828125$ memory_percent‚88.58346939086914 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.dm-1.in‚0.5625 network.sent‚ 2.2761467e+07# disk.nvme0c0n1.in‚ 1755.46484375 cpu‚25.32275427499577 disk.rbd3.out‚0* proc.memory.percent‚16.632914543151855) disk./.usagePercent‚74.40182994799893 disk.rbd0.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2725.13671875 disk.rbd0.out‚0 disk.rbd2.out‚ 613.900390625 disk.sda1.in‚0 network.recv‚ 2.825376e+06% disk.nvme0c0n1.out‚66073.98828125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1755.46484375% disk./.usageGB‚1108.6410331726074 disk.sda3.in‚0.5625 disk.sda.out‚278.3125 disk.dm-0.out‚66073.98828125 disk.sda3.out‚278.3125 disk.dm-0.in‚ 1755.46484375æÖœ”°#‚(:‰ Æ„æÔïžÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚255.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W˜ãÌC±#‚(:¸ Æ„æÔè£ÝÊ disk.sda1.out‚0( disk.nvme0c0n1.out‚66171.57470703125 proc.cpu.threads‚17 disk.sda2.in‚0 cpu‚25.322226480119003! disk.rbd2.out‚614.03369140625 disk.sda.out‚ 278.71484375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46484375) disk./.usagePercent‚74.40182482784824# disk.nvme0c0n1.in‚ 1755.46484375$ memory_percent‚88.58060836791992 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0 network.sent‚ 2.2766694e+07 network.recv‚ 2.829091e+06$ disk./.usageGB‚1108.640956878662 disk.rbd3.in‚0& disk.nvme0n1.out‚66171.57470703125# disk.dm-0.out‚66171.57470703125 disk.dm-1.out‚ 278.71484375* proc.memory.percent‚16.632914543151855 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 278.71484375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.6796875' proc.memory.availableMB‚ 1870.953125# proc.memory.rssMB‚ 2725.13671875 disk.sda.in‚0.5625Á}¢Å2²#- _wandbruntime‚7695  _runtime‚7695‰År·Æ³#À & lr-AdamW/pg2‚6.786768268587518e-05 trainer/global_step‚9099 $ _timestamp‚1.788445255744573e+09 _runtime‚7695.789240852 _step‚745 & lr-AdamW/pg1‚6.786768268587518e-05›æcË´#Å _step‚745 & lr-AdamW/pg1‚6.786768268587518e-05 & lr-AdamW/pg2‚6.786768268587518e-05 trainer/global_step‚9099 $ _timestamp‚1.788445255744573e+09 _runtime‚7695.789240852é:¼Èß2µ#- _wandbruntime‚7695  _runtime‚7695ÆÅZ(¶#”P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87183066 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚9099 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16884525417984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1031698910197248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1014814384779264 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150285883 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚7695.790264509 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134359411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63102367 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚891241172189696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16887125627392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304108301 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150285433 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16884525417984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134359405 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚998608830258176 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169748890 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1014814411170816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169728856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1031698910197248 % _timestamp‚1.7884452557453594e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚998608855686144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1031701536798208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868120560950272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1015465363560034 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1014814411170816 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚891241246720512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23120639373824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16887125627392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87183069 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868120532815872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16859131586526 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134359411 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1014814384779264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16856533301858ü¨Š(·#™P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚891241246720512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23120639373824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16887125627392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87183069 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868120532815872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16859131586526 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134359411 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1014814384779264 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16856533301858 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87183066 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚9099 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16884525417984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1031698910197248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1014814384779264 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150285883 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚7695.790264509 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134359411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63102367 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚891241172189696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16887125627392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304108301 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150285433 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16884525417984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134359405 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚998608830258176 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169748890 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1014814411170816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169728856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1031698910197248 % _timestamp‚1.7884452557453594e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚998608855686144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1031701536798208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868120560950272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1015465363560034 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1014814411170816 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960êÈT$—2¸#- _wandbruntime‚7696  _runtime‚7696 DãÑm!¹#N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304120731 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚868212949003776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134373647 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚998715171432448 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169767150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63108970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134373641 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23123215716352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1031811308845568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1031808682231296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134373641 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1031811308845568 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150301232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1014922324484608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150301665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16886357746688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16860960929402 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87192265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚891336088105472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_tra6 èжin_batch_end/allocation.small_pool.peak‚24124 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1014922324484608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 7696.53988963 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140797 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16888957969408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63109400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1015576157789818 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920Ëõõ:('º#¢N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169767150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63108970 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134373641 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23123215716352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1031811308845568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1031808682231296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134373641 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1031811308845568 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150301232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1014922324484608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150301665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16886357746688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16860960929402 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87192265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚891336088105472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1014922324484608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 7696.53988963 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304140797 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16888957969408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63109400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1015576157789818 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304120731 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚868212949003776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134373647 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚998715171432448 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ë¼#½Ý2»#- _wandbruntime‚7696  _runtime‚7696"ÒY\¼#Ö 2loss/SRqqt2/assignment_loss‚1.2918338033816088 2loss/SRqqt4/assignment_loss‚0.9015429303316134 epoch‚7 /loss/FBt2/assignment_loss‚0.539105117165794 /loss/FRt4/detection_loss‚2.2882882431149483 1loss/SRqqt1/detection_loss‚0.9616811540031779 /loss/FBt3/detection_loss‚2.1843678997113156 /loss/FBt1/detection_loss‚0.6092522413359562 0loss/FRt2/assignment_loss‚2.3329934683713045 0loss/FRt4/assignment_loss‚2.5006857936580977 1loss/FBt1/assignment_loss‚0.08690081372982333 1loss/FBt4/assignment_loss‚0.38418965943492367 /loss/FRt1/detection_loss‚1.6614297510717402 .loss/FRt3/detection_loss‚2.857196885604747 1loss/SRqqt2/detection_loss‚1.8346920904258024 /loss/FRt1/assignment_loss‚1.185607450929555 0loss/FRt3/assignment_loss‚3.6917943399527977 .loss/FRt2/detection_loss‚2.123559978875247 1loss/SRqqt4/detection_loss‚1.9120071640326863 &loss/total_loss‚40.454109034913536 trainer/global_step‚9099 2loss/SRqqt3/assignment_loss‚1.4717925230581914 _runtime‚7696.541007384 _step‚748 0loss/SRqqt3/detection_loss‚2.314253589799327 /loss/FBt2/detection_loss‚3.4245104789733887 .loss/FBt4/detection_loss‚2.884047731757164 1loss/FBt3/assignment_loss‚0.03381701668191594 2loss/SRqqt1/assignment_loss‚0.9785589095124084 $ _timestamp‚1.788445256496413e+09íãÂÓa½#Û 0loss/SRqqt3/detection_loss‚2.314253589799327 /loss/FBt2/detection_loss‚3.4245104789733887 .loss/FBt4/detection_loss‚2.884047731757164 1loss/FBt3/assignment_loss‚0.03381701668191594 2loss/SRqqt1/assignment_loss‚0.9785589095124084 $ _timestamp‚1.788445256496413e+09 2loss/SRqqt2/assignment_loss‚1.2918338033816088 2loss/SRqqt4/assignment_loss‚0.9015429303316134 epoch‚7 /loss/FBt2/assignment_loss‚0.539105117165794 /loss/FRt4/detection_loss‚2.2882882431149483 1loss/SRqqt1/detection_loss‚0.9616811540031779 /loss/FBt3/detection_loss‚2.1843678997113156 /loss/FBt1/detection_loss‚0.6092522413359562 0loss/FRt2/assignment_loss‚2.3329934683713045 0loss/FRt4/assignment_loss‚2.5006857936580977 1loss/FBt1/assignment_loss‚0.08690081372982333 1loss/FBt4/assignment_loss‚0.38418965943492367 /loss/FRt1/detection_loss‚1.6614297510717402 .loss/FRt3/detection_loss‚2.857196885604747 1loss/SRqqt2/detection_loss‚1.8346920904258024 /loss/FRt1/assignment_loss‚1.185607450929555 0loss/FRt3/assignment_loss‚3.6917943399527977 .loss/FRt2/detection_loss‚2.123559978875247 1loss/SRqqt4/detection_loss‚1.9120071640326863 &loss/total_loss‚40.454109034913536 trainer/global_step‚9099 2loss/SRqqt3/assignment_loss‚1.4717925230581914 _runtime‚7696.541007384 _step‚748ìn%†’¾#‚(:‡ Õ„æÔƒ¡¼É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vóG#?¿#‚(:´ Õ„æÔŒÖÂË disk.dm-1.out‚ 279.3046875 disk.drbd1777.out‚0 disk.rbd1.out‚3.828125 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-0.in‚ 1755.46484375 cpu‚25.32172557798665 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚66172.51318359375 network.sent‚ 2.2772568e+07( disk./.usagePercent‚74.3729364256695! disk.nvme0n1.in‚ 1755.46484375( disk.nvme0c0n1.out‚66172.51318359375* proc.memory.percent‚16.632914543151855 network.recv‚ 2.833023e+06 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0# disk.dm-0.out‚66172.51318359375$ memory_percent‚88.57836723327637 disk.dm-1.in‚0.5625 disk.rbd3.out‚0% disk./.usageGB‚1108.2104988098145! disk.rbd2.out‚614.04931640625 disk.sda1.in‚0( proc.memory.availableMB‚ 1871.3203125 disk.sda.out‚ 279.3046875 disk.sda2.in‚0 disk.sda3.out‚ 279.3046875‡aïÏ”À#‚(:‰ ä„æÔƒ¨šÉ gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—){[CÁ#‚(:¸ ä„æÔó†¶Ë% disk./.usageGB‚1108.2104988098145 disk.dm-0.in‚ 1755.46484375 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚66269.96630859375 disk.sda3.out‚ 279.88671875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.in‚0$ memory_percent‚88.57769966125488 disk.sda2.out‚0 network.recv‚ 2.839514e+06 disk.dm-1.out‚ 279.88671875! disk.rbd2.out‚614.18994140625 disk.sda.in‚0.5625 network.sent‚ 2.2881511e+07 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.632914543151855 cpu‚25.321200472782586( proc.memory.availableMB‚ 1871.4296875( disk.nvme0c0n1.out‚66269.96630859375# disk.dm-0.out‚66269.96630859375 disk.dm-1.in‚0.5625 disk.rbd1.out‚3.828125 disk.sda.out‚ 279.88671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0.5625( disk./.usagePercent‚74.3729364256695# disk.nvme0c0n1.in‚ 1755.46484375 disk.drbd1777.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2725.136718756–ì£2Â#- _wandbruntime‚7736  _runtime‚7736ݟ;ÄÃ#¾ _step‚749 % lr-AdamW/pg1‚6.57546086105769e-05 % lr-AdamW/pg2‚6.57546086105769e-05 trainer/global_step‚9149 $ _timestamp‚1.788445295983049e+09 _runtime‚7736.027742023ƒÚî*ÉÄ#à trainer/global_step‚9149 $ _timestamp‚1.788445295983049e+09 _runtime‚7736.027742023 _step‚749 % lr-AdamW/pg1‚6.57546086105769e-05 % lr-AdamW/pg2‚6.57546086105769e-05íÍW†¹2Å#- _wandbruntime‚7736  _runtime‚7736²%Ô(Æ#–P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87642869 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63438033 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚895986548940288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16950612311134 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1037187526313472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚7736.028721185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1020211370046464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _step‚750 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16976156267008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚895986623471104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1020876526710878 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135071205 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚872739963625472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170663942 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1037190152914432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135071211 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135071211 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87642866 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1037187526313472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135071205 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63437577 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170643908 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305735153 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1020211396438016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16948014026466 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16976156267008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1020211370046464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151080443 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003925888971776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1020211396438016 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1020873902998242 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Wé• QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305735153 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1020211396438016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16948014026466 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16976156267008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1020211370046464 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151080443 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003925888971776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1020211396438016 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1020873902998242 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1037187526313472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚7736.028721185 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1020211370046464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _step‚750 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16976156267008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074îO£ÊÇ2È#- _wandbruntime‚7736  _runtime‚7736óÑ ›&'É# N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1037297298347520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87652062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23249174588928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020984697228026 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170682202 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚872832351678976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16949843369134 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1037299924961792 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151096414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63444352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1004032230146048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135085447 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16952441654010 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1020319309751808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135085441 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1004032255574016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16980588818432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16977988595712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1020319336143360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170682202 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151096863 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305767649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1020319309751808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135085447 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16977988595712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚896081449653248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚872832323544576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884452967341342e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 _runtime‚7736.779680423 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020982073515182 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1020319336143360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170662142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63444798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 trainer/global_step‚9149 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135085447 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16977988595712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚896081449653248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚872832323544576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884452967341342e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 _runtime‚7736.779680423 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020982073515182 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1020319336143360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170662142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63444798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 trainer/global_step‚9149 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020984697228026 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170682202 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚872832351678976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16949843369134 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1037299924961792 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151096414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63444352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1004032230146048 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135085447 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16952441654010 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1020319309751808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135085441 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1004032255574016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16980588818432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16977988595712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1020319336143360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ï;ÔS2Ë#- _wandbruntime‚7736  _runtime‚77368Û:SÌ#Í 0loss/FRt2/assignment_loss‚2.7064803613556756 0loss/FRt3/assignment_loss‚3.2760884181873218 2loss/SRqqt1/assignment_loss‚0.9027347244220237 1loss/SRqqt1/detection_loss‚0.9630946897174397 .loss/FBt2/detection_loss‚2.447509467601776 /loss/FRt1/assignment_loss‚1.033757912615935 /loss/FBt3/assignment_loss‚0.633422534330748 .loss/FRt2/detection_loss‚2.450064590242174 /loss/FRt3/detection_loss‚2.2603347470222444 1loss/SRqqt4/detection_loss‚1.7119563586182065 .loss/FBt4/detection_loss‚2.701992188181196 2loss/SRqqt3/assignment_loss‚1.6123800029357274 ,loss/FRt1/detection_loss‚1.3688472417028 _runtime‚7736.780733394 /loss/FRt4/assignment_loss‚2.684439683953921 2loss/SRqqt4/assignment_loss‚1.3603578522387478 /loss/FBt1/assignment_loss‚0.117026059696946 .loss/FRt4/detection_loss‚2.510096641878287 .loss/FBt1/detection_loss‚0.603597948776691 /loss/FBt3/detection_loss‚2.5794291496276855 %loss/total_loss‚39.39302153789709 trainer/global_step‚9149 1loss/FBt2/assignment_loss‚0.11112361904812929 epoch‚7 _step‚752 1loss/FBt4/assignment_loss‚0.10611815831673864 1loss/SRqqt2/assignment_loss‚1.249464104654359 1loss/SRqqt2/detection_loss‚1.6845514290645474 1loss/SRqqt3/detection_loss‚2.3181536537077694 $ _timestamp‚1.788445296736144e+09˜«ªXXÍ#Ò 1loss/SRqqt2/assignment_loss‚1.249464104654359 1loss/SRqqt2/detection_loss‚1.6845514290645474 1loss/SRqqt3/detection_loss‚2.3181536537077694 $ _timestamp‚1.788445296736144e+09 0loss/FRt2/assignment_loss‚2.7064803613556756 0loss/FRt3/assignment_loss‚3.2760884181873218 2loss/SRqqt1/assignment_loss‚0.9027347244220237 1loss/SRqqt1/detection_loss‚0.9630946897174397 .loss/FBt2/detection_loss‚2.447509467601776 /loss/FRt1/assignment_loss‚1.033757912615935 /loss/FBt3/assignment_loss‚0.633422534330748 .loss/FRt2/detection_loss‚2.450064590242174 /loss/FRt3/detection_loss‚2.2603347470222444 1loss/SRqqt4/detection_loss‚1.7119563586182065 .loss/FBt4/detection_loss‚2.701992188181196 2loss/SRqqt3/assignment_loss‚1.6123800029357274 ,loss/FRt1/detection_loss‚1.3688472417028 _runtime‚7736.780733394 /loss/FRt4/assignment_loss‚2.684439683953921 2loss/SRqqt4/assignment_loss‚1.3603578522387478 /loss/FBt1/assignment_loss‚0.117026059696946 .loss/FRt4/detection_loss‚2.510096641878287 .loss/FBt1/detection_loss‚0.603597948776691 /loss/FBt3/detection_loss‚2.5794291496276855 %loss/total_loss‚39.39302153789709 trainer/global_step‚9149 1loss/FBt2/assignment_loss‚0.11112361904812929 epoch‚7 _step‚752 1loss/FBt4/assignment_loss‚0.10611815831673864ð’Ÿ*MžÎ#‚(:“ ó„æÔà°©É gpu.0.gpu‚88.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.068' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.21359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹†ôß2Ï#‚(:§ ó„æÔÖÜÔË disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1755.46484375 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2725.13671875 proc.cpu.threads‚17 network.recv‚ 2.844017e+06 disk.rbd1.in‚0 disk.sda.in‚0.5625$ memory_percent‚88.57765197753906* proc.memory.percent‚16.632914543151855% disk./.usageGB‚1108.2121887207031! disk.rbd2.out‚614.20166015625" disk.nvme0n1.out‚ 66272.3671875 disk.dm-1.out‚ 280.5234375% proc.memory.availableMB‚ 1871.4375 network.sent‚ 2.2887745e+07 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.dm-0.out‚ 66272.3671875 disk.dm-1.in‚0.5625 cpu‚25.320675085824266 disk.sda1.out‚0! disk.nvme0n1.in‚ 1755.46484375 disk.sda.out‚ 280.5234375 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1755.46484375$ disk.nvme0c0n1.out‚ 66272.3671875 disk.rbd1.out‚3.890625 disk.sda1.in‚0 disk.sda3.out‚ 280.5234375) disk./.usagePercent‚74.37304983700716 disk.drbd1777.in‚0‹¹(Õ”Ð#‚(:‰ ‚…æÔÁØœÉ gpu.0.gpu‚73.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*æw7Ñ#‚(:¬ ‚…æÔëÁøÊ) disk./.usagePercent‚74.37305239708252# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.in‚0 disk.rbd1.out‚3.890625# disk.nvme0n1.out‚66272.70703125 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625# proc.memory.rssMB‚ 2725.13671875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.2996164e+07 disk.dm-1.out‚ 281.0234375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.632914543151855% disk./.usageGB‚1108.2122268676758! disk.rbd2.out‚615.90087890625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 281.0234375( proc.memory.availableMB‚ 1871.5546875 cpu‚25.32015826086842% disk.nvme0c0n1.out‚66272.70703125 disk.sda.in‚0.5625 disk.dm-0.in‚ 1755.46484375 disk.dm-0.out‚66272.70703125 disk.sda1.in‚0$ memory_percent‚88.57693672180176! disk.nvme0n1.in‚ 1755.46484375 network.recv‚ 2.850091e+06 disk.sda.out‚ 281.0234375 disk.rbd3.in‚0UFt¬”Ò#‚(:‰ ‘…æÔ‚£¼É gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i“ªÛ Ó#‚(:º ‘…æÔå¼ÁË* proc.memory.percent‚16.632914543151855! disk.nvme0n1.in‚ 1755.46484375& disk.nvme0n1.out‚66370.38330078125# disk.nvme0c0n1.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 281.7265625 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1108.2122535705566 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚88.57729434967041 cpu‚25.31963464067539 network.sent‚ 2.3002161e+07 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625) proc.memory.availableMB‚ 1871.49609375 disk.dm-0.in‚ 1755.46484375# disk.dm-0.out‚66370.38330078125 disk.sda.out‚ 281.7265625 disk.sda3.out‚ 281.7265625 disk.sda2.in‚0 disk.sdaHr<;2.out‚0 disk.rbd3.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2725.13671875( disk.nvme0c0n1.out‚66370.38330078125 network.recv‚ 2.854386e+06 disk.rbd0.in‚0 disk.rbd1.out‚ 3.9228515625) disk./.usagePercent‚74.37305418913526 disk.sda3.in‚0.5625! disk.rbd2.out‚615.91650390625eôØ2Ô#- _wandbruntime‚7776  _runtime‚7776Ð~¿wÇÕ#Á trainer/global_step‚9199 % _timestamp‚1.7884453365959072e+09 _runtime‚7776.640713549 _step‚753 & lr-AdamW/pg1‚6.366821005158978e-05 & lr-AdamW/pg2‚6.366821005158978e-05±¤ÌÖ#Æ _runtime‚7776.640713549 _step‚753 & lr-AdamW/pg1‚6.366821005158978e-05 & lr-AdamW/pg2‚6.366821005158978e-05 trainer/global_step‚9199 % _timestamp‚1.7884453365959072e+09ñhm’2×#- _wandbruntime‚7776  _runtime‚7776õ],Û(Ø#–P H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1025608381705216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135783005 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23372269655040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17070387325440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341965 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1025608355313664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17067787116032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171578994 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1042676142429696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1025608355313664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚900731635955712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚877359338166272 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135783011 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151877901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17070387325440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1026285066149086 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1025608381705216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88102669 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚754 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171558960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17067787116032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1042676142429696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚900731559327744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171578994 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1042678769030656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135783011 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63775702 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1042678769030656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63775235 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7776.641769897 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23372221161472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17042093035742 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171558960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151878371 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1009242973113344 trainer/global_step‚9199ÄùЇ!(Ù#›P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1026285066149086 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1025608381705216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88102669 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚754 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171558960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17067787116032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1042676142429696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚900731559327744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171578994 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1042678769030656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135783011 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63775702 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1042678769030656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63775235 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7776.641769897 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23372221161472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17042093035742 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171558960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151878371 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1009242973113344 trainer/global_step‚9199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1025608381705216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135783005 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23372269655040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17070387325440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341965 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1025608355313664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17067787116032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171578994 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1042676142429696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1025608355313664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚900731635955712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚877359338166272 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135783011 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151877901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17070387325440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400ò:íAþ2Ú#- _wandbruntime‚7777  _runtime‚7777—Eæ¡&'Û# N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026393236666234 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1026390612953390 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151893741 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1009349314287616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63782324 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307374435 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135797241 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17069619444736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1025716295019008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17041324093742 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171577194 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17043922378618 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚7777.407227992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151894189 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63781879 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚900826538339840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1025716321410560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1009349288859648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88111862 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1042788541078016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171577194 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1042788541078016 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1042785914463744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171597254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88111865 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877451754354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23374735505408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1009349288859648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88111862 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1042788541078016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171577194 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_p£KZu8"ool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1042788541078016 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1042785914463744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171597254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88111865 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877451754354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23374735505408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151893741 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1009349314287616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63782324 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307374435 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135797241 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17069619444736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1025716295019008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17041324093742 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171577194 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17043922378618 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚7777.407227992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720óÁ,+ä2Ý#- _wandbruntime‚7777  _runtime‚7777×ƘàXÞ#Ò 0loss/FBt1/assignment_loss‚0.1263387903856423 1loss/SRqqt4/detection_loss‚1.7012930721678632 .loss/FBt2/detection_loss‚2.533114016056061 trainer/global_step‚9199 0loss/FRt4/assignment_loss‚2.1310860574245454 2loss/SRqqt4/assignment_loss‚0.8110358216997949 /loss/FRt1/detection_loss‚1.6217847614670577 .loss/FRt4/detection_loss‚2.191072877049446 _runtime‚7777.408366093 _step‚756 /loss/FRt2/assignment_loss‚2.713369380235672 0loss/FBt4/assignment_loss‚0.1577543043709334 2loss/SRqqt2/assignment_loss‚1.6001224754021524 1loss/SRqqt2/detection_loss‚1.9678333376316315 1loss/FBt3/assignment_loss‚0.11846205533947796 .loss/FBt3/detection_loss‚2.192822331190109 %loss/total_loss‚36.64972286473302 % _timestamp‚1.7884453373637938e+09 /loss/FRt1/assignment_loss‚0.990050714948903 2loss/SRqqt1/assignment_loss‚0.8367293234543361 1loss/SRqqt3/assignment_loss‚1.298179309294363 -loss/FRt3/detection_loss‚2.27059913652389 1loss/SRqqt1/detection_loss‚0.7985335620849029 0loss/FBt1/detection_loss‚0.46141816558706206 .loss/FBt4/detection_loss‚2.401565041806963 0loss/FRt3/assignment_loss‚2.9178495278229586 1loss/FBt2/assignment_loss‚0.29831061789688346 /loss/FRt2/detection_loss‚2.2989357867836953 0loss/SRqqt3/detection_loss‚2.211462398108683 epoch‚7tÄúJ]ß#× 0loss/SRqqt3/detection_loss‚2.211462398108683 epoch‚7 0loss/FBt1/assignment_loss‚0.1263387903856423 1loss/SRqqt4/detection_loss‚1.7012930721678632 .loss/FBt2/detection_loss‚2.533114016056061 trainer/global_step‚9199 0loss/FRt4/assignment_loss‚2.1310860574245454 2loss/SRqqt4/assignment_loss‚0.8110358216997949 /loss/FRt1/detection_loss‚1.6217847614670577 .loss/FRt4/detection_loss‚2.191072877049446 _runtime‚7777.408366093 _step‚756 /loss/FRt2/assignment_loss‚2.713369380235672 0loss/FBt4/assignment_loss‚0.1577543043709334 2loss/SRqqt2/assignment_loss‚1.6001224754021524 1loss/SRqqt2/detection_loss‚1.9678333376316315 1loss/FBt3/assignment_loss‚0.11846205533947796 .loss/FBt3/detection_loss‚2.192822331190109 %loss/total_loss‚36.64972286473302 % _timestamp‚1.7884453373637938e+09 /loss/FRt1/assignment_loss‚0.990050714948903 2loss/SRqqt1/assignment_loss‚0.8367293234543361 1loss/SRqqt3/assignment_loss‚1.298179309294363 -loss/FRt3/detection_loss‚2.27059913652389 1loss/SRqqt1/detection_loss‚0.7985335620849029 0loss/FBt1/detection_loss‚0.46141816558706206 .loss/FBt4/detection_loss‚2.401565041806963 0loss/FRt3/assignment_loss‚2.9178495278229586 1loss/FBt2/assignment_loss‚0.29831061789688346 /loss/FRt2/detection_loss‚2.2989357867836953ô|°)Ÿà#‚(:”  …æÔØ×äÉ gpu.0.gpu‚51.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èMÓƒ<á#‚(:±  …æÔþ‰§Ë! disk.rbd2.out‚615.99462890625 network.sent‚ 2.3007971e+07# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625$ memory_percent‚88.57893943786621# proc.memory.rssMB‚ 2725.13671875 disk.dm-1.out‚ 282.3359375# disk.nvme0n1.out‚66370.78515625 network.recv‚ 2.858959e+06% disk./.usageGB‚1108.2122192382812 disk.dm-0.in‚ 1755.46484375 disk.dm-0.out‚66370.78515625 disk.dm-1.in‚0.5625 disk.sda.out‚ 282.3359375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 3.9228515625 disk.sda1.out‚0* proc.memory.percent‚16.632914543151855 disk.sda3.out‚ 282.3359375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚74.37305188506744 disk.sda3.in‚0.5625% disk.nvme0c0n1.out‚66370.78515625 disk.drbd1777.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1755.46484375( proc.memory.availableMB‚ 1871.2265625 proc.cpu.threads‚17 disk.rbd0.in‚0 cpu‚25.319118296285065Bþ7Š“â#‚(:ˆ ¯…æÔŒÉ’Ê gpu.0.gpu‚67.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚249.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜nãÓBã#‚(:· ¯…æÔ„¼ýÊ disk.drbd1777.out‚0$ memory_percent‚88.57824802398682 disk.dm-1.out‚ 282.73046875 network.sent‚ 2.3117327e+07$ disk./.usageGB‚1108.212345123291! disk.dm-0.out‚66468.693359375$ disk.nvme0n1.out‚66468.693359375 disk.sda3.in‚0.5625 disk.rbd2.out‚616.0185546875 disk.sda.in‚0.5625 disk.sda2.in‚0! disk.nvme0n1.in‚ 1755.46484375# proc.memory.rssMB‚ 2725.13671875 cpu‚25.318604751052604 network.recv‚ 2.865488e+06& disk.nvme0c0n1.out‚66468.693359375 disk.sda1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚74.37306033331608 disk.drbd1777.in‚0 disk.sda.out‚ 282.73046875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1871.33984375# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.in‚0 disk.rbd1.out‚ 3.98583984375 disk.rbd2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.632914543151855 proc.cpu.threads‚17 disk.sda3.out‚ 282.73046875 disk.dm-0.in‚ 1755.46484375 disk.dm-1.in‚0.5625;¬”ä#‚(:‰ ¾…æÔ̺É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚117.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r øÅBå#‚(:· ¾…æÔ ú¹Ë disk.sda3.in‚0.5625 disk.rbd2.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.632914543151855% disk.nvme0n1.out‚66469.0849609375 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚616.0185546875 disk.sda2.in‚0# memory_percent‚88.5767936706543 proc.cpu.threads‚17' disk.nvme0c0n1.out‚66469.0849609375 disk.dm-1.out‚ 283.47265625 disk.rbd1.out‚ 3.98583984375 disk.sda.in‚0.5625 disk.rbd3.in‚0' proc.memory.availableMB‚ 1871.578125! disk.nvme0n1.in‚ 1755.46484375# disk.nvme0c0n1.in‚ 1755.46484375 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2725.13671875 cpu‚25.318119630748285 network.sent‚ 2.312255e+07 network.recv‚ 2.869266e+06) disk./.usagePercent‚74.37304087674347 disk.rbd0.out‚0" disk.dm-0.out‚66469.0849609375 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0% disk./.usageGB‚1108.2120552062988 disk.sda3.out‚ 283.47265625 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46484375 disk.sda.out‚ 283.47265625P©E 2æ#- _wandbruntime‚7817  _runtime‚7817ÓžÇç#Á & lr-AdamW/pg1‚6.160892181124645e-05 & lr-AdamW/pg2‚6.160892181124645e-05 trainer/global_step‚9249 % _timestamp‚1.7884453772643576e+09 _runtime‚7817.309063434 _step‚757V á€Ìè#Æ & lr-AdamW/pg2‚6.160892181124645e-05 trainer/global_step‚9249 % _timestamp‚1.7884453772643576e+09 _runtime‚7817.309063434 _step‚757 & lr-AdamW/pg1‚6.160892181124645e-05õµȱ2é#- _wandbruntime‚7817  _runtime‚7817;u+†(ê#–P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88562469 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152673026 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚905476778245120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23498009269248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1031693605587294 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚758 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881978740841472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1031005366972416 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031005340580864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136494811 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136494811 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1014560031826944 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136494805 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1031690981874658 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88562466 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17133573760350 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172494046 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1048164758545920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172474012 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1014560006398976 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1031005340580864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152672562 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 trainer/global_step‚9249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17159417965056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172494046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905476699520000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚7817.310036406 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17162018174464 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172474012 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64110557 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1048164758545920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1031005366972416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64110096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1031005366972416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64110096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152673026 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚905476778245120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23498009269248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1031693605587294 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚758 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881978740841472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1031005366972416 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031005340580864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136494811 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136494811 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1014560031826944 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136494805 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1031690981874658 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88562466 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17133573760350 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172494046 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1048164758545920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649ö™ñC‚2ì#- _wandbruntime‚7818  _runtime‚7818ZÒ7&'í# N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882071128894976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 trainer/global_step‚9249 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1048277157194240 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1031113306677760 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1048274530579968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172512306 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17163850516480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 % _timestamp‚1.7884453780244954e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1048277157194240 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17163850516480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001287 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136509041 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1031113280286208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1031799152391598 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309021353 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64117167 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136509041 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172492246 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152688391 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64116729 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚905571665162752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1031113280286208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23500508133376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1031113306677760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152688832 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1014666347573248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17132804818350 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚7818.069701796 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136509047 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88571665 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚905571588548096 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1031801776104442 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376Êðïù+'î#¥N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1048277157194240 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17163850516480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001287 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136509041 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1031113280286208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1031799152391598 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309021353 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64117167 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136509041 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172492246 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152688391 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64116729 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚905571665162752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1031113280286208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23500508133376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1031113306677760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152688832 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1014666347573248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17132804818350 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚7818.069701796 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136509047 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88571665 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚905571588548096 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1031801776104442 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882071128894976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 trainer/global_step‚9249 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1048277157194240 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1031113306677760 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1048274530579968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172512306 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17163850516480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 % _timestamp‚1.7884453780244954e+09÷í†ÚÈ2ï#- _wandbruntime‚7818  _runtime‚7818z…#]ð#× $ _timestamp‚1.788445378026428e+09 _step‚760 2loss/SRqqt1/assignment_loss‚0.8207623921056737 1loss/FBt2/assignment_loss‚0.03656545667520403 .loss/FRt3/detection_loss‚2.427284359551933 trainer/global_step‚9249 /loss/FRt2/assignment_loss‚2.565241737799211 0loss/FRt3/assignment_loss‚3.4425086396081106 1loss/FBt1/assignment_loss‚0.10004107199887406 1loss/SRqqt1/detection_loss‚0.8303140809120082 0loss/FBt1/detection_loss‚0.44798673525581073 .loss/FBt2/detection_loss‚2.635613203048706 _runtime‚7818.071067206 2loss/SRqqt3/assignment_loss‚1.4409834418607794 3loss/FBt4/assignment_loss‚0.0069787366956006736 1loss/SRqqt2/detection_loss‚1.9126971899108454 1loss/SRqqt3/detection_loss‚2.0360005621028985 2loss/SRqqt2/assignment_loss‚1.4070153300735084 0loss/FRt1/assignment_loss‚1.0303616436819236 /loss/FRt4/detection_loss‚2.4153795259035364 /loss/FBt4/detection_loss‚2.2818001906077066 1loss/FBt3/assignment_loss‚0.05314444373860689 -loss/FRt2/detection_loss‚2.53238840197975 1loss/SRqqt4/detection_loss‚1.7740371975426872 epoch‚7 /loss/FRt4/assignment_loss‚2.280588924884796 /loss/FRt1/detection_loss‚1.7129122542294983 2loss/SRqqt4/assignment_loss‚1.2388997693390895 .loss/FBt3/detection_loss‚2.524798001561846 $loss/total_loss‚37.9543032910686Áñø`bñ#Ü /loss/FBt4/detection_loss‚2.2818001906077066 1loss/FBt3/assignment_loss‚0.05314444373860689 -loss/FRt2/detection_loss‚2.53238840197975 1loss/SRqqt4/detection_loss‚1.7740371975426872 epoch‚7 /loss/FRt4/assignment_loss‚2.280588924884796 /loss/FRt1/detection_loss‚1.7129122542294983 2loss/SRqqt4/assignment_loss‚1.2388997693390895 .loss/FBt3/detection_loss‚2.524798001561846 $loss/total_loss‚37.9543032910686 $ _timestamp‚1.788445378026428e+09 _step‚760 2loss/SRqqt1/assignment_loss‚0.8207623921056737 1loss/FBt2/assignment_loss‚0.03656545667520403 .loss/FRt3/detection_loss‚2.427284359551933 trainer/global_step‚9249 /loss/FRt2/assignment_loss‚2.565241737799211 0loss/FRt3/assignment_loss‚3.4425086396081106 1loss/FBt1/assignment_loss‚0.10004107199887406 1loss/SRqqt1/detection_loss‚0.8303140809120082 0loss/FBt1/detection_loss‚0.44798673525581073 .loss/FBt2/detection_loss‚2.635613203048706 _runtime‚7818.071067206 2loss/SRqqt3/assignment_loss‚1.4409834418607794 3loss/FBt4/assignment_loss‚0.0069787366956006736 1loss/SRqqt2/detection_loss‚1.9126971899108454 1loss/SRqqt3/detection_loss‚2.0360005621028985 2loss/SRqqt2/assignment_loss‚1.4070153300735084 0loss/FRt1/assignment_loss‚1.0303616436819236 /loss/FRt4/detection_loss‚2.4153795259035364øs¶õŸò#‚(:” Í…æÔ¢‡É gpu.0.gpu‚91.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚267.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.479600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¦åÙBó#‚(:· Í…æÔÛÇ¥Ë disk.rbd0.out‚0( disk./.usagePercent‚74.3730552131654 disk.rbd3.in‚0 disk.sda3.in‚0.5625* proc.memory.percent‚16.632914543151855 cpu‚25.31760882759831" disk.dm-0.out‚66566.7587890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 284.08203125) proc.memory.availableMB‚ 1871.40234375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1755.46484375 disk.drbd1777.out‚0 disk.sda.out‚ 284.08203125 network.recv‚ 2.873595e+06% disk./.usageGB‚1108.2122688293457' disk.nvme0c0n1.out‚66566.7587890625 disk.drbd1777.in‚0 disk.rbd2.in‚0$ memory_percent‚88.57786655426025 network.sent‚ 2.3128546e+07 disk.rbd1.out‚ 4.048828125 disk.sda.in‚0.5625 proc.cpu.threads‚17 disk.rbd1.in‚0% disk.nvme0n1.out‚66566.7587890625# proc.memory.rssMB‚ 2725.13671875 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 284.08203125 disk.rbd2.out‚617.6083984375/¿C“ô#‚(:ˆ Ü…æÔ¿µýÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚93.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó^§:Bõ#‚(:· Ü…æÔ°åÑË disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.632914543151855 network.recv‚ 2.880473e+06 disk.rbd2.in‚0 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1755.46484375 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd1.out‚ 4.048828125 disk.rbd3.out‚0 disk.sda3.in‚0.5625 network.sent‚ 2.3237388e+07" disk.dm-0.out‚66567.0673828125 disk.rbd3.in‚0) proc.memory.availableMB‚ 1871.64453125 disk.rbd2.out‚617.6083984375 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚25.3171283587975% disk.nvme0n1.out‚66567.0673828125% disk./.usageGB‚1108.2122116088867 disk.dm-1.in‚0.5625 disk.sda1.out‚0 disk.drbd1777.out‚0$ memory_percent‚88.57638835906982 disk.dm-1.out‚ 284.52734375 disk.sda.out‚ 284.52734375 disk.sda3.out‚ 284.52734375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2725.13671875) disk./.usagePercent‚74.37305137305238' disk.nvme0c0n1.out‚66567.0673828125 disk.dm-0.in‚ 1755.46484375"ÙO2ö#- _wandbruntime‚7858  _runtime‚7858¨êãsÉ÷#à ' lr-AdamW/pg1‚5.9577173042129625e-05 ' lr-AdamW/pg2‚5.9577173042129625e-05 trainer/global_step‚9299 % _timestamp‚1.7884454180979428e+09 _runtime‚7858.142687808 _step‚761ý4ìÎø#È ' lr-AdamW/pg1‚5.9577173042129625e-05 ' lr-AdamW/pg2‚5.9577173042129625e-05 trainer/global_step‚9299 % _timestamp‚1.7884454180979428e+09 _runtime‚7858.142687808 _step‚761ùÇu³2ù#- _wandbruntime‚7858  _runtime‚7858Óu"ú#–P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1036402325848064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153467942 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1053653374662144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17251048814080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64446144 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚886598143516672 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1037099521312866 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17225054484958 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173409098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17253649023488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173389064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310595669 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1036402325848064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 trainer/global_step‚9299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17251048814080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310615709 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1019877065112576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1053656001263104 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1019877090540544 _step‚762 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚471 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17222456200290 % _timestamp‚1.7884454180988128e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23623525876224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137206605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64445676 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0\W“î!(û#›P KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173389064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137206611 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚886598171651072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚7858.143784293 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1036402352239616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137206611 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89022269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1053656001263104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1036402352239616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1053653374662144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17251048814080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64446144 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚886598143516672 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1037099521312866 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17225054484958 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173409098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17253649023488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173389064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310595669 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1036402325848064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 trainer/global_step‚9299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17251048814080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310615709 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1019877065112576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1053656001263104 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1019877090540544 _step‚762 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚471 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17222456200290 % _timestamp‚1.7884454180988128e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23623525876224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137206605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64445676 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1036402325848064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153467942 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1019983431714816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23626044925440 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173407298 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17252881142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886690559704576 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173407298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884454188633208e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137220847 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64452402 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚910316525918208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1053763146696192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1019983406286848 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1053765773310464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137220841 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886690531570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137220847 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17224285542958 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64452852 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _runtime‚7858.908520777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1036510291944960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648205 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310628139 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1053765773310464 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1037207691829806 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1036510265553408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137220841 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173427358 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1037210315542650 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17252881142784 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153484317 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153483864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89031465 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1019983431714816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23626044925440 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173407298 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17252881142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886690559704576 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173407298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884454188633208e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137220847 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64452402 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_trai•W…¤Án_batch_end/allocation.all.allocated‚ 310648205 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚910316525918208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1053763146696192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1019983406286848 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1053765773310464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137220841 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886690531570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137220847 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17224285542958 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64452852 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _runtime‚7858.908520777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1036510291944960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310648205 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310628139 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1053765773310464 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1037207691829806 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1036510265553408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137220841 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173427358 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1037210315542650 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17252881142784 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153484317 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153483864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89031465 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174304116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89482069 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89482066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚891217574326272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1041799337506816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174304116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154263000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17345279872512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137918405 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1041799311115264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚914966609747456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1042510684463710 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17345279872512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 trainer/global_step‚9349 _runtime‚7899.169658616 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137918405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚766 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1059141990778368 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1041799337506816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 " _timestamp‚1.7884454591247e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174324150 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚891217546191872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137918411 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1059144617379328 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312242561 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137918411 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64780931 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23749114146304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776¥ávª@!$˜P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1042510684463710 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17345279872512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 trainer/global_step‚9349 _runtime‚7899.169658616 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137918405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚766 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1059141990778368 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1041799337506816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 " _timestamp‚1.7884454591247e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174324150 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚891217546191872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137918411 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1059144617379328 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312242561 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137918411 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64780931 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23749114146304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17342679663104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174304116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89482069 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 330301PŸôÇÞ4400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89482066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚891217574326272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1041799337506816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174304116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154263000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17345279872512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137918405 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1041799311115264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚914966609747456þÈb¾í2Ž$- _wandbruntime‚7899  _runtime‚7899•k:m&'$ N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚891309962379776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89491265 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64787616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1025300490428416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89491262 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1041907277212160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1041907250820608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17315766267566 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1041907250820608 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 _runtime‚7899.939656632 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174342410 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1042616231268014 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137932647 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137932641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174322350 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1041907277212160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚891309934245376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137932641 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174322350 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17347112214528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17344511991808 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1025300465000448 trainer/global_step‚9349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚915061589806592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1059254389426688 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1059251762812416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137932647 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154278881 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚915061511094784 _step‚767 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23751627426816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154278433 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17347112214528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174342410 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254991 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137932647 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154278881 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚915061511094784 _step‚767 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23751627426816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154278433 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17347112214528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174342410 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254991 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89491265 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64787616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1025300490428416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89491262 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1041907277212160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1041907250820608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17315766267566 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1041907250820608 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 _runtime‚7899.939656632 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174342410 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1042616231268014 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137932647 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137932641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174322350 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1041907277212160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚891309934245376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ÿPŒÎ2‘$- _wandbruntime‚7899  _runtime‚7899Àæwf^’$Ø /loss/FBt2/detection_loss‚2.3516665480353613 1loss/FBt3/assignment_loss‚0.19797666636683667 /loss/FRt1/detection_loss‚1.7277908283285797 /loss/FRt2/assignment_loss‚2.354785075554481 1loss/SRqqt3/assignment_loss‚1.618151978072193 2loss/SRqqt4/assignment_loss‚1.1908531474044262 .loss/FRt3/detection_loss‚2.754571599620876 1loss/FBt4/assignment_loss‚0.33152346898402485 /loss/FBt3/detection_loss‚3.3526213069756827 %loss/total_loss‚38.30741486742464 epoch‚7 _step‚768 0loss/FRt4/assignment_loss‚1.6733854247464075 1loss/FBt2/assignment_loss‚0.15153704667781395 /loss/FBt4/detection_loss‚2.1651941537857056 _runtime‚7899.940877991 0loss/FRt1/assignment_loss‚0.9817621386051179 /loss/FRt2/detection_loss‚2.6556509404610367 1loss/SRqqt1/detection_loss‚1.1416534158504672 trainer/global_step‚9349 % _timestamp‚1.7884454598962376e+09 0loss/FRt3/assignment_loss‚3.0432037752250145 2loss/SRqqt1/assignment_loss‚0.9365807315140652 1loss/SRqqt2/detection_loss‚1.9074625538289547 /loss/FBt1/detection_loss‚0.5246289449373298 2loss/SRqqt2/assignment_loss‚1.2884311653673648 .loss/FRt4/detection_loss‚2.039694505157294 0loss/SRqqt4/detection_loss‚1.582425308934713 1loss/FBt1/assignment_loss‚0.18867259447785467 1loss/SRqqt3/detection_loss‚2.1471915485130415—r\c“$Ý /loss/FBt2/detection_loss‚2.3516665480353613 1loss/FBt3/assignment_loss‚0.19797666636683667 /loss/FRt1/detection_loss‚1.7277908283285797 /loss/FRt2/assignment_loss‚2.354785075554481 1loss/SRqqt3/assignment_loss‚1.618151978072193 2loss/SRqqt4/assignment_loss‚1.1908531474044262 .loss/FRt3/detection_loss‚2.754571599620876 1loss/FBt4/assignment_loss‚0.33152346898402485 /loss/FBt3/detection_loss‚3.3526213069756827 %loss/total_loss‚38.30741486742464 epoch‚7 _step‚768 0loss/FRt4/assignment_loss‚1.6733854247464075 1loss/FBt2/assignment_loss‚0.15153704667781395 /loss/FBt4/detection_loss‚2.1651941537857056 _runtime‚7899.940877991 0loss/FRt1/assignment_loss‚0.9817621386051179 /loss/FRt2/detection_loss‚2.6556509404610367 1loss/SRqqt1/detection_loss‚1.1416534158504672 trainer/global_step‚9349 % _timestamp‚1.7884454598962376e+09 0loss/FRt3/assignment_loss‚3.0432037752250145 2loss/SRqqt1/assignment_loss‚0.9365807315140652 1loss/SRqqt2/detection_loss‚1.9074625538289547 /loss/FBt1/detection_loss‚0.5246289449373298 2loss/SRqqt2/assignment_loss‚1.2884311653673648 .loss/FRt4/detection_loss‚2.039694505157294 0loss/SRqqt4/detection_loss‚1.582425308934713 1loss/FBt1/assignment_loss‚0.18867259447785467 1loss/SRqqt3/detection_loss‚2.1471915485130415€9º-Ÿ”$‚(:” ˜†æÔˆ‰žÊ gpu.0.gpu‚72.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚211.133' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.226600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XHn¬=•$‚(:² ˜†æÔº®¤Ë network.recv‚ 2.899709e+06( disk.nvme0c0n1.out‚66763.38720703125 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.sda3.in‚0.5625( proc.memory.availableMB‚ 1871.0859375 disk.dm-0.in‚ 1755.46484375! disk.nvme0n1.in‚ 1755.46484375 disk.rbd0.in‚0 disk.dm-1.out‚ 286.19140625 disk.sda1.out‚0& disk.nvme0n1.out‚66763.38720703125* proc.memory.percent‚16.632938385009766# disk.nvme0c0n1.in‚ 1755.46484375 disk.rbd2.in‚0! proc.memory.rssMB‚ 2725.140625 disk.drbd1777.out‚0 disk.rbd2.out‚ 617.859375 disk.sda.out‚ 286.19140625 disk.sda3.out‚ 286.19140625 cpu‚25.3150785212382$ disk./.usageGB‚1108.212532043457 disk.rbd1.out‚ 4.048828125) disk./.usagePercent‚74.37307287768526# disk.dm-0.out‚66763.38720703125 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0$ memory_percent‚88.57979774475098 network.sent‚ 2.3363849e+07 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17;RäR”–$‚(:‰ §†æÔó€®É gpu.0.gpu‚89.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚252.683' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.5366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EFÝ0—$‚(:¥ §†æÔ°ãÓÌ disk.rbd1.out‚ 4.048828125 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1755.46484375 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0$ memory_percent‚88.58215808868408 network.sent‚ 2.3473515e+07$ disk./.usageGB‚1108.212589263916 disk.rbd3.in‚0& disk.nvme0n1.out‚66763.90185546875) proc.memory.availableMB‚ 1870.69921875) disk./.usagePercent‚74.37307671779827 disk.sda.out‚286.75 disk.rbd0.in‚0* proc.memory.percent‚16.632938385009766 cpu‚25.314569018674426 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1755.46484375 proc.cpu.threads‚17 disk.sda3.out‚286.75 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 619.43359375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1755.46484375# disk.dm-0.out‚66763.90185546875! proc.memory.rssMB‚ 2725.140625 disk.rbd2.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚66763.90185546875 disk.dm-1.out‚286.75 network.recv‚ 2.906438e+06 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0Çãô˜$‚(:’ ¶†æÔÊ×ùÈ gpu.0.gpu‚27.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚37.0 gpu.0.powerWatts‚89.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.874399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰k+5™$‚(:ª ¶†æÔøÍ°Ë disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1755.46484375" disk.dm-0.out‚66861.4521484375 disk.dm-1.out‚ 287.03125 disk.sda3.out‚ 287.03125) disk./.usagePercent‚74.37307671779827 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1870.94140625! proc.memory.rssMB‚ 2725.140625% disk.nvme0n1.out‚66861.4521484375 network.sent‚ 2.347874e+07 disk.rbd1.out‚ 4.048828125 disk.sda.out‚ 287.03125 disk.sda3.in‚0.5625 disk.rbd2.out‚ 619.44921875! disk.nvme0n1.in‚ 1755.46484375$ memory_percent‚88.58067989349365 cpu‚25.314079129808853 disk.sda2.in‚0 network.recv‚ 2.910288e+06$ disk./.usageGB‚1108.212589263916 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.632938385009766# disk.nvme0c0n1.in‚ 1755.46484375' disk.nvme0c0n1.out‚66861.4521484375LæÁ2š$- _wandbruntime‚7939  _runtime‚7939ôEDûÆ›$À trainer/global_step‚9399 % _timestamp‚1.7884454997258332e+09 _runtime‚ 7939.77056365 _step‚769 & lr-AdamW/pg1‚5.559798174374527e-05 & lr-AdamW/pg2‚5.559798174374527e-05ZwŸËœ$Å & lr-AdamW/pg1‚5.559798174374527e-05 & lr-AdamW/pg2‚5.559798174374527e-05 trainer/global_step‚9399 % _timestamp‚1.7884454997258332e+09 _runtime‚ 7939.77056365 _step‚769·'«2$- _wandbruntime‚7939  _runtime‚7939ñFLqÇ ž$•P _step‚770 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1064630606894592 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1047919223901918 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1047196322774016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1047196296382464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17434310512128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1064630606894592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚919711259526144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175239202 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175239202 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175219168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23874282524672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚919711180801024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17434310512128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23874231933952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17405417649506 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_bô©’Tatch_start/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138630211 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1064633233495552 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1047196322774016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17436910721536 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17408015934174 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138630211 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚9399 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155060501 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175219168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1064630606894592 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1047919223901918 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1047196322774016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1047196296382464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17434310512128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1064630606894592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚919711259526144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175239202 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175239202 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175219168 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23874282524672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚919711180801024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17434310512128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23874231933952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17405417649506 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138630211 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1064633233495552 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1047196322774016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17436910721536 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17408015934174 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138630211 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 trainer/global_step‚9399 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155060501 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175219168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23876743780352 trainer/global_step‚9399 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1047304236087808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚919806080700928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884455004799438e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138644447 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175237402 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65125863 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138644441 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17436142840832 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1064743005542912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1030617523714048 _runtime‚ 7940.5252689 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1064743005542912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17409845277050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚771 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1047304262479360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175257462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1048027394419066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚895929365054976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138644447 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155076472 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1047304262479360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17438743063552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65125410 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1048024770706222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚895929336920576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17407246992174 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 180931624960ªóÍ¢$£N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89951065 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65125863 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138644441 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17436142840832 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1064743005542912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1030617523714048 _runtime‚ 7940.5252689 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1064743005542912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17409845277050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚771 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1047304262479360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175257462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1048027394419066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚895929365054976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138644447 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155076472 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1047304262479360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17438743063552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65125410 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1048024770706222 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚895929336920576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17407246992174 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23876743780352 trainer/global_step‚9399 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1047304236087808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚919806080700928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884455004799438e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138644447 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175237402 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1070119223010816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 trainer/global_step‚9449 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176154254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924455665100800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176134220 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17525941361152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139342005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65455581 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139342005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚924455586375680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23999234833408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1035828241253376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17499496658782 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1052593308041216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1035828266681344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚7980.343646831 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65456064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176134220 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1052593281649664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 trainer/global_step‚9449 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176154254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924455665100800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176134220 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17525941361152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139342005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65455581 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139342005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚924455586375680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23999234833408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1035828241253376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17499496658782 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1052593308041216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1035828266681344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚7980.343646831 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65456064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176134220 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1052593281649664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1070119223010816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139356241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924550564369408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1052701247746560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1052701247746560 _step‚775 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90410862 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139356247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1053433310144430 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 $ _timestamp‚1.788445541052694e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1052701221355008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155873041 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65462625 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1035934582427648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 _runtime‚ 7981.09778407 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17530373912576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚900548767730176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24001796639232 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1035934607855616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17501326001658 trainer/global_step‚9449 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176172514 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0…s )'´$£N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1052701247746560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1052701247746560 _step‚775 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90410862 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139356247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1053433310144430 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 $ _timestamp‚1.788445541052694e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1052701221355008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155873041 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65462625 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1035934582427648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 _runtime‚ 7981.09778407 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17530373912576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚900548767730176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24001796639232 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1035934607855616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17501326001658 trainer/global_step‚9449 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176172514 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1052701221355008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17498727716782 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1070231621659136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17527773689856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65462179 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508695 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90410865 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚900548739595776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24001746061824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17530373912576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17527773689856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139356247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1070228995044864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176152454 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139356241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924550564369408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649‡ÌøZw2µ$- _wandbruntime‚7981  _runtime‚7981 ¡[¶$Õ 1loss/FBt4/assignment_loss‚0.12582122469029855 /loss/FRt1/detection_loss‚1.6746158610262416 1loss/SRqqt3/assignment_loss‚1.443968765599572 1loss/SRqqt1/detection_loss‚0.9129838054733617 0loss/SRqqt3/detection_loss‚2.009126867937005 _step‚776 /loss/FRt4/detection_loss‚2.2482612587987107 trainer/global_step‚9449 /loss/FRt1/assignment_loss‚1.164239067657321 2loss/SRqqt2/assignment_loss‚1.1685912224407098 1loss/FBt2/assignment_loss‚0.07109606261365116 1loss/FBt3/assignment_loss‚0.19330813689076523 /loss/FRt3/detection_loss‚2.8095645904541016 1loss/SRqqt2/detection_loss‚1.8827481336739598 2loss/SRqqt1/assignment_loss‚0.8730224560946226 &loss/total_loss‚37.587998897132934 /loss/FRt2/assignment_loss‚2.739010514439763 /loss/FRt4/assignment_loss‚2.404227651180105 2loss/SRqqt4/assignment_loss‚1.0803561244848305 _runtime‚7981.099103654 /loss/FRt3/assignment_loss‚2.409022322472404 .loss/FRt2/detection_loss‚2.530803198749955 1loss/SRqqt4/detection_loss‚1.6791720206484846 /loss/FBt2/detection_loss‚2.6532858431339266 -loss/FBt3/detection_loss‚2.43034870756997 epoch‚7 1loss/FBt1/assignment_loss‚0.24360398789057974 /loss/FBt1/detection_loss‚0.7097520724022096 /loss/FBt4/detection_loss‚2.1310690008103848 % _timestamp‚1.7884455410545042e+09#ˆ?‹`·$Ú -loss/FBt3/detection_loss‚2.43034870756997 epoch‚7 1loss/FBt1/assignment_loss‚0.24360398789057974 /loss/FBt1/detection_loss‚0.7097520724022096 /loss/FBt4/detection_loss‚2.1310690008103848 % _timestamp‚1.7884455410545042e+09 1loss/FBt4/assignment_loss‚0.12582122469029855 /loss/FRt1/detection_loss‚1.6746158610262416 1loss/SRqqt3/assignment_loss‚1.443968765599572 1loss/SRqqt1/detection_loss‚0.9129838054733617 0loss/SRqqt3/detection_loss‚2.009126867937005 _step‚776 /loss/FRt4/detection_loss‚2.2482612587987107 trainer/global_step‚9449 /loss/FRt1/assignment_loss‚1.164239067657321 2loss/SRqqt2/assignment_loss‚1.1685912224407098 1loss/FBt2/assignment_loss‚0.07109606261365116 1loss/FBt3/assignment_loss‚0.19330813689076523 /loss/FRt3/detection_loss‚2.8095645904541016 1loss/SRqqt2/detection_loss‚1.8827481336739598 2loss/SRqqt1/assignment_loss‚0.8730224560946226 &loss/total_loss‚37.587998897132934 /loss/FRt2/assignment_loss‚2.739010514439763 /loss/FRt4/assignment_loss‚2.404227651180105 2loss/SRqqt4/assignment_loss‚1.0803561244848305 _runtime‚7981.099103654 /loss/FRt3/assignment_loss‚2.409022322472404 .loss/FRt2/detection_loss‚2.530803198749955 1loss/SRqqt4/detection_loss‚1.6791720206484846 /loss/FBt2/detection_loss‚2.6532858431339266ˆ”éÌ”¸$‚(:‰ ò†æÔƒŒàÊ gpu.0.gpu‚68.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚230.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.1152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä÷ý=¹$‚(:² ò†æÔÙÔƒÍ disk.rbd2.in‚0 disk.sda3.out‚ 289.19921875! disk.dm-0.out‚66960.591796875 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚66960.591796875 disk.dm-1.in‚0.5625* proc.memory.percent‚16.632938385009766 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.5625# disk.nvme0c0n1.in‚ 1755.46484375! proc.memory.rssMB‚ 2725.140625) disk./.usagePercent‚74.37314686386264 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1108.2136344909668 disk.rbd1.out‚ 4.048828125 disk.sda3.in‚0.5625& disk.nvme0c0n1.out‚66960.591796875 disk.dm-1.out‚ 289.19921875 cpu‚25.31212426531889 disk.drbd1777.in‚0 disk.sda.out‚ 289.19921875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1755.46484375! disk.nvme0n1.in‚ 1755.46484375$ memory_percent‚88.58084678649902 network.sent‚ 2.3604216e+07 network.recv‚ 2.928759e+06 disk.rbd2.out‚621.2119140625( proc.memory.availableMB‚ 1870.9140625 proc.cpu.threads‚17s»Ñ”º$‚(:‰ ‡æÔϳÉ gpu.0.gpu‚71.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž¬¬<»$‚(:± ‡æÔ©åœÌ disk.rbd2.in‚0 disk.dm-0.in‚ 1755.46484375 disk.dm-1.out‚ 289.84375 disk.rbd3.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1755.46484375! proc.memory.rssMB‚ 2725.140625# disk.dm-0.out‚67058.19384765625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1755.46484375* proc.memory.percent‚16.632938385009766 network.recv‚ 2.935336e+06 disk.rbd1.in‚0 disk.sda3.out‚ 289.84375 disk.rbd0.in‚0 disk.rbd1.out‚ 4.048828125 disk.sda.in‚0.5625 disk.sda.out‚ 289.84375 disk.rbd3.out‚0% disk./.usageGB‚1108.2136917114258 disk.rbd2.out‚621.2744140625& disk.nvme0n1.out‚67058.19384765625( disk.nvme0c0n1.out‚67058.19384765625$ memory_percent‚88.58110904693604 proc.cpu.threads‚17 network.sent‚ 2.3713248e+07 disk.dm-1.in‚0.5625 disk.sda1.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1870.87109375) disk./.usagePercent‚74.37315070397567 disk.sda1.in‚0 disk.sda3.in‚0.5625 cpu‚25.311640333078923ã…ÿ2¼$- _wandbruntime‚8020  _runtime‚8020ªîɽ$à ' lr-AdamW/pg1‚5.1733953013630266e-05 ' lr-AdamW/pg2‚5.1733953013630266e-05 trainer/global_step‚9499 % _timestamp‚1.7884455807311783e+09 _runtime‚8020.775869206 _step‚777‹Ô±Üξ$È ' lr-AdamW/pg1‚5.1733953013630266e-05 ' lr-AdamW/pg2‚5.1733953013630266e-05 trainer/global_step‚9499 % _timestamp‚1.7884455807311783e+09 _runtime‚8020.775869206 _step‚777‰u”µ2¿$- _wandbruntime‚8020  _runtime‚8020+½Ë~À$•P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1041145325394944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24124408556032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1058733679065698 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177069306 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177049272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90861466 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140053805 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156652321 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1075610465728000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1058736302778334 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140053811 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140053811 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17620172419584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156652796 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚905075754217472 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1075607839127040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317123117 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚929200162773504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1057990293308416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _runtime‚8020.776905787 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17590977383390 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚905075782351872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104114sw!5299966976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177069306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65790855 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1075607839127040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17617572210176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1075610465728000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17620172419584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1057990266916864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177049272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚929200241498624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1057990266916864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17617572210176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140053805 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 trainer/global_step‚9499 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24124408556032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1058733679065698 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177069306 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177049272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90861466 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140053805 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156652321 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1075610465728000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1058736302778334 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140053811 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140053811 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17620172419584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156652796 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚905075754217472 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1075607839127040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317123117 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚929200162773504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1057990293308416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _runtime‚8020.776905787 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17590977383390 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚905075782351872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1041145299966976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177069306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65790855 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1075607839127040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17617572210176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1075610465728000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17620172419584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1057990266916864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177049272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚929200241498624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1057990266916864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17617572210176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776ŠÖrí¤2Â$- _wandbruntime‚8021  _runtime‚8021øoÏó&'Ã$ N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚929295060442112 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1075720237775360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1075717611161088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058841849582638 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24126966651392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884455814994485e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90870665 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1041251641141248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚929295137056768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17592806726266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140068047 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17619404538880 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚905168170405376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140068041 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177067506 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1058098206622208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140068041 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140068047 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1058098206622208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17619404538880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177067506 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚905168142270976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17590208441390 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156668275 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1041251666569216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17622004761600Œ ŒÈÄ$¥N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058841849582638 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24126966651392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884455814994485e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90870665 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1041251641141248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚929295137056768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17592806726266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140068047 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17619404538880 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚905168170405376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140068041 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177067506 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1058098206622208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140068041 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140068047 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1058098206622208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17619404538880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177067506 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚905168142270976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17590208441390 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peûÀ…cak‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156668275 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1041251666569216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17622004761600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚929295060442112 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1075720237775360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1075717611161088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‹-³‡¾2Å$- _wandbruntime‚8021  _runtime‚8021¬ò™\Æ$Ö 1loss/SRqqt1/detection_loss‚0.7448663187878473 _step‚780 /loss/FRt2/assignment_loss‚2.120553847621469 1loss/SRqqt3/assignment_loss‚1.562686441956382 /loss/FBt3/assignment_loss‚0.882294217008166 /loss/FRt2/detection_loss‚2.7194050559226204 /loss/FRt3/detection_loss‚2.3398001614492387 .loss/FRt4/detection_loss‚2.396545779705048 0loss/FBt1/detection_loss‚0.46148637107490925 2loss/SRqqt4/assignment_loss‚1.5142332615204535 &loss/total_loss‚40.003018648617385 0loss/SRqqt2/detection_loss‚2.077230440960689 1loss/SRqqt4/detection_loss‚1.4190055859922353 2loss/SRqqt2/assignment_loss‚1.4311890094964341 /loss/FBt2/detection_loss‚2.6345360815525054 /loss/FBt4/detection_loss‚3.6352458238601684 % _timestamp‚1.7884455815013838e+09 trainer/global_step‚9499 1loss/FBt1/assignment_loss‚0.06112635733119992 .loss/FBt3/detection_loss‚2.074953990323203 0loss/FRt1/assignment_loss‚0.9963942449143592 0loss/FRt3/assignment_loss‚3.4407583862543105 1loss/FBt2/assignment_loss‚0.19644275933969765 1loss/FBt4/assignment_loss‚0.15587102719582618 epoch‚7 _runtime‚8021.546003504 /loss/FRt4/assignment_loss‚2.482991099357605 /loss/FRt1/detection_loss‚1.6252425285650696 0loss/SRqqt3/detection_loss‚2.059991717338562 2loss/SRqqt1/assignment_loss‚0.9701681410893798zè˜aÇ$Û 0loss/FRt1/assignment_loss‚0.9963942449143592 0loss/FRt3/assignment_loss‚3.4407583862543105 1loss/FBt2/assignment_loss‚0.19644275933969765 1loss/FBt4/assignment_loss‚0.15587102719582618 epoch‚7 _runtime‚8021.546003504 /loss/FRt4/assignment_loss‚2.482991099357605 /loss/FRt1/detection_loss‚1.6252425285650696 0loss/SRqqt3/detection_loss‚2.059991717338562 2loss/SRqqt1/assignment_loss‚0.9701681410893798 1loss/SRqqt1/detection_loss‚0.7448663187878473 _step‚780 /loss/FRt2/assignment_loss‚2.120553847621469 1loss/SRqqt3/assignment_loss‚1.562686441956382 /loss/FBt3/assignment_loss‚0.882294217008166 /loss/FRt2/detection_loss‚2.7194050559226204 /loss/FRt3/detection_loss‚2.3398001614492387 .loss/FRt4/detection_loss‚2.396545779705048 0loss/FBt1/detection_loss‚0.46148637107490925 2loss/SRqqt4/assignment_loss‚1.5142332615204535 &loss/total_loss‚40.003018648617385 0loss/SRqqt2/detection_loss‚2.077230440960689 1loss/SRqqt4/detection_loss‚1.4190055859922353 2loss/SRqqt2/assignment_loss‚1.4311890094964341 /loss/FBt2/detection_loss‚2.6345360815525054 /loss/FBt4/detection_loss‚3.6352458238601684 % _timestamp‚1.7884455815013838e+09 trainer/global_step‚9499 1loss/FBt1/assignment_loss‚0.06112635733119992 .loss/FBt3/detection_loss‚2.074953990323203ŒÏƒu’È$‚(:‡ ‡æÔ¼ÌÙË gpu.0.gpu‚75.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚120.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.188 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ߢœu/É$‚(:¤ ‡æÔíǦÎ network.sent‚ 2.3719111e+07# disk.nvme0n1.out‚67058.48828125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 1755.46875 disk.sda.out‚ 290.3984375) proc.memory.availableMB‚ 1870.77734375 cpu‚25.31117029876595) disk./.usagePercent‚74.38157411987477 disk.sda1.in‚0! proc.memory.rssMB‚ 2725.140625 disk.rbd0.in‚0 disk.dm-0.in‚ 1755.46875 disk.dm-0.out‚67058.48828125 disk.rbd2.out‚621.2900390625% disk./.usageGB‚1108.3392066955566 disk.sda1.out‚0 disk.nvme0n1.in‚ 1755.46875% disk.nvme0c0n1.out‚67058.48828125 disk.rbd1.in‚0$ memory_percent‚88.58168125152588* proc.memory.percent‚16.632938385009766 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.dm-1.out‚ 290.3984375 disk.drbd1777.out‚0 disk.sda.in‚0.5625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 290.3984375 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 4.048828125 network.recv‚ 2.939934e+06 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0iè¼>ŸÊ$‚(:” Ÿ‡æÔúëûÈ gpu.0.gpu‚44.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚115.264' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.052799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›~63Ë$‚(:¨ Ÿ‡æÔÁÕçË) proc.memory.availableMB‚ 1870.54296875* proc.memory.percent‚16.632938385009766 cpu‚25.31067007549739 disk.rbd1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚74.38153955885764 disk.drbd1777.in‚0 disk.sda.out‚ 290.96875 disk.sda2.in‚0 network.sent‚ 2.3828362e+07 disk.drbd1777.out‚0 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚67059.22216796875% disk./.usageGB‚1108.3386917114258 disk.rbd1.out‚ 4.048828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 290.96875! disk.rbd2.out‚621.37646484375 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.sda3.out‚ 290.96875 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 1755.46875 network.recv‚ 2.946458e+06# disk.dm-0.out‚67059.22216796875 disk.dm-1.in‚0.5625! proc.memory.rssMB‚ 2725.140625 proc.cpu.threads‚17 disk.dm-0.in‚ 1755.46875 disk.sda1.in‚0 disk.nvme0n1.in‚ 1755.46875& disk.nvme0n1.out‚67059.22216796875$ memory_percent‚88.58311176300049}ãóÆžÌ$‚(:“ ®‡æÔÏø¨É gpu.0.gpu‚55.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.77759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñý0Í7Í$‚(:¬ ®‡æÔçÇ›Ë disk.rbd0.out‚0! disk.rbd2.out‚621.38818359375 disk.rbd3.out‚0& proc.memory.availableMB‚ 1870.78125" disk.dm-0.out‚67156.8369140625 network.sent‚ 2.3833702e+07 disk.nvme0c0n1.in‚ 1755.46875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 4.048828125 disk.sda.out‚ 291.76171875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚67156.8369140625 disk.dm-1.in‚0.5625 disk.sda2.out‚0 network.recv‚ 2.950272e+06% disk./.usageGB‚1108.3389854431152 disk.sda.in‚0.5625% disk.nvme0n1.out‚67156.8369140625 disk.dm-1.out‚ 291.76171875 disk.drbd1777.out‚0 disk.sda3.out‚ 291.76171875 cpu‚25.310190070254244) disk./.usagePercent‚74.38155927143778 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1755.46875 disk.drbd1777.in‚0* proc.memory.percent‚16.632938385009766 disk.nvme0n1.in‚ 1755.46875$ memory_percent‚88.58165740966797! proc.memory.rssMB‚ 2725.140625 disk.sda2.in‚0 disk.sda1.in‚0ˆtÌ2Î$- _wandbruntime‚8062  _runtime‚8062Ð%tÆÏ$À & lr-AdamW/pg1‚4.984613495520348e-05 & lr-AdamW/pg2‚4.984613495520348e-05 trainer/global_step‚9549 $ _timestamp‚1.788445622164557e+09 _runtime‚8062.209306172 _step‚781cB‡¢ËÐ$Å & lr-AdamW/pg1‚4.984613495520348e-05 & lr-AdamW/pg2‚4.984613495520348e-05 trainer/global_step‚9549 $ _timestamp‚1.788445622164557e+09 _runtime‚8062.209306172 _step‚781þ62Ñ$- _wandbruntime‚8062  _runtime‚8062%I%(Ò$–P ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1064144842216542 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17709203059200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17709203059200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66126829 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1046462358680576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157448579 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚933944734409728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1081096455243264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063387278575616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063387252184064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177964324 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24249498791936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177964324 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 % _timestamp‚1.7884456221654217e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚909695156892672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140765611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140765605 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17711803268608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1081099081844224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177984358 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91321266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1081099081844224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140765605 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140765611 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1081096455243264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672ª/´}!(Ó$›P H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 % _timestamp‚1.7884456221654217e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚909695156892672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140765611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140765605 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17711803268608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1081099081844224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177984358 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91321266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1081099081844224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140765605 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140765611 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1081096455243264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1064144842216542 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17709203059200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17709203059200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚481 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66126829 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1046462358680576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157448579 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚933944734409728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1081096455243264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063387278575616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063387252184064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177964324 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24249498791936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177964324 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074ŽŒUé2Ô$- _wandbruntime‚8062  _runtime‚8062ç¬É„dÕ$ N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66134085 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1063495191889408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚909787544946176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17713635610624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66133614 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140779841 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178002618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚934039624598528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24252000940544 % _timestamp‚1.7884456229390426e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17711035387904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚934039545886720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚909787573080576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚9549 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177982558 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063495218280960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1046568699854848 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17684287450874 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91330462 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140779847 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17711035387904 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063495218280960 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24252051517952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1064250389020846 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1081208853891584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157464076 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1081206227277312 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063495191889408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91330465 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17713635610624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17681689165998 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140779847 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1064253012733690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚783 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248;Ñ–S+'Ö$¥N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1046568699854848 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17684287450874 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91330462 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140779847 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17711035387904 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063495218280960 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24252051517952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1064250389020846 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1081208853891584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157464076 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1081206227277312 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063495191889408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91330465 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17713635610624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17681689165998 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140779847 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1064253012733690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚783 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66134085 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1063495191889408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚909787544946176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17713635610624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66133614 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140779841 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178002618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚934039624598528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24252000940544 % _timestamp‚1.7884456229390426e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17711035387904 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚934039545886720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚909787573080576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚9549 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177982558 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063495218280960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496øêÌ£2×$- _wandbruntime‚8062  _runtime‚8062â–=UØ$Ï 0loss/FRt1/assignment_loss‚0.9704786563913027 .loss/FRt4/detection_loss‚2.448796733127286 1loss/SRqqt1/detection_loss‚0.8065701529267244 1loss/SRqqt2/detection_loss‚1.7116857065508764 % _timestamp‚1.7884456229407673e+09 1loss/SRqqt1/assignment_loss‚0.714395600320131 2loss/SRqqt4/assignment_loss‚1.3401492567449484 1loss/SRqqt4/detection_loss‚1.7341732690515725 /loss/FBt1/detection_loss‚0.5097211884852085 .loss/FBt2/detection_loss‚2.149371168830178 _step‚784 /loss/FRt2/assignment_loss‚2.702719517376112 1loss/FBt2/assignment_loss‚0.26103917763049883 .loss/FRt2/detection_loss‚2.244825179486171 /loss/FBt4/detection_loss‚1.7557246685028076 2loss/SRqqt2/assignment_loss‚1.1464828809645649 /loss/FRt1/detection_loss‚1.3989534236801167 /loss/FRt3/assignment_loss‚3.116596619784832 1loss/FBt4/assignment_loss‚0.15267285289750857 1loss/SRqqt3/detection_loss‚2.4024577607711155 $loss/total_loss‚39.5183917888182 .loss/FRt3/detection_loss‚2.044471912004519 2loss/SRqqt3/assignment_loss‚1.5503073501090208 epoch‚7 trainer/global_step‚9549 _runtime‚8062.985384862 +loss/FRt4/assignment_loss‚ 3.03039966772 1loss/FBt1/assignment_loss‚0.15649804933967032 1loss/FBt3/assignment_loss‚0.15273686511742804 .loss/FBt3/detection_loss‚5.017164131005605ëw¯ŽZÙ$Ô 1loss/SRqqt1/detection_loss‚0.8065701529267244 1loss/SRqqt2/detection_loss‚1.7116857065508764 % _timestamp‚1.7884456229407673e+09 1loss/SRqqt1/assignment_loss‚0.714395600320131 2loss/SRqqt4/assignment_loss‚1.3401492567449484 1loss/SRqqt4/detection_loss‚1.7341732690515725 /loss/FBt1/detection_loss‚0.5097211884852085 .loss/FBt2/detection_loss‚2.149371168830178 _step‚784 /loss/FRt2/assignment_loss‚2.702719517376112 1loss/FBt2/assignment_loss‚0.26103917763049883 .loss/FRt2/detection_loss‚2.244825179486171 /loss/FBt4/detection_loss‚1.7557246685028076 2loss/SRqqt2/assignment_loss‚1.1464828809645649 /loss/FRt1/detection_loss‚1.3989534236801167 /loss/FRt3/assignment_loss‚3.116596619784832 1loss/FBt4/assignment_loss‚0.15267285289750857 1loss/SRqqt3/detection_loss‚2.4024577607711155 $loss/total_loss‚39.5183917888182 .loss/FRt3/detection_loss‚2.044471912004519 2loss/SRqqt3/assignment_loss‚1.5503073501090208 epoch‚7 trainer/global_step‚9549 _runtime‚8062.985384862 +loss/FRt4/assignment_loss‚ 3.03039966772 1loss/FBt1/assignment_loss‚0.15649804933967032 1loss/FBt3/assignment_loss‚0.15273686511742804 .loss/FBt3/detection_loss‚5.017164131005605 0loss/FRt1/assignment_loss‚0.9704786563913027 .loss/FRt4/detection_loss‚2.448796733127286ƒ0ƒeŸÚ$‚(:” ½‡æÔáÀšÉ gpu.0.gpu‚83.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚205.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.044799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È‚äÞ2Û$‚(:§ ½‡æÔ¤ôóË disk.sda1.out‚0 disk.sda3.out‚ 292.35546875 disk.dm-1.out‚ 292.35546875* proc.memory.percent‚16.632938385009766 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚67157.93115234375# disk.dm-0.out‚67157.93115234375 network.sent‚ 2.3839592e+07% disk./.usageGB‚1108.3389854431152 disk.dm-0.in‚ 1755.46875) disk./.usagePercent‚74.38155927143778! disk.rbd2.out‚621.45068359375" memory_percent‚88.580322265625& disk.nvme0n1.out‚67157.93115234375 disk.nvme0c0n1.in‚ 1755.46875 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.048828125 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 1755.46875 disk.sda3.in‚0.5625 disk.rbd0.out‚0 proc.memory.availableMB‚1871! proc.memory.rssMB‚ 2725.140625 cpu‚25.309703597867173 network.recv‚ 2.954522e+06 disk.sda.out‚ 292.35546875X"Pc’Ü$‚(:‡ ̇æÔôŠ¼É gpu.0.gpu‚89.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¶ÝR<Ý$‚(:± ̇æÔ—®ÑË disk.rbd2.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚67255.49072265625 disk.rbd1.out‚ 4.048828125& disk.nvme0n1.out‚67255.49072265625 disk.rbd0.in‚0# disk.dm-0.out‚67255.49072265625 cpu‚25.30919425767969 disk.dm-1.out‚ 292.81640625 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 292.81640625) disk./.usagePercent‚74.38155543132477! disk.rbd2.out‚623.04052734375 disk.sda3.in‚0.5625* proc.memory.percent‚16.632938385009766 disk.sda.out‚ 292.81640625 network.recv‚ 2.961256e+06 disk.dm-1.in‚0.5625 disk.sda2.in‚0! proc.memory.rssMB‚ 2725.140625 disk.rbd3.in‚0 disk.dm-0.in‚ 1755.46875$ memory_percent‚88.58177661895752 proc.cpu.threads‚17% disk./.usageGB‚1108.3389282226562 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 1755.46875 disk.rbd0.out‚0) proc.memory.availableMB‚ 1870.76171875 network.sent‚ 2.3948529e+07 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚ 1755.46875¶cb`”Þ$‚(:‰ Û‡æÔʳÁÉ gpu.0.gpu‚73.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚156.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“BñÃ6ß$‚(:« Û‡æÔÔ”²Ë disk.rbd3.out‚0) disk./.usagePercent‚74.38187825682544 disk.sda2.out‚0 disk.nvme0n1.in‚ 1755.46875 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.sda.out‚ 293.390625 disk.sda1.out‚0! proc.memory.rssMB‚ 2725.140625 disk.nvme0c0n1.in‚ 1755.46875 disk.dm-1.out‚ 293.390625 disk.sda2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 293.390625# disk.dm-0.out‚67261.60595703125 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.048828125& disk.nvme0n1.out‚67261.60595703125 network.recv‚ 2.965721e+06 disk.dm-0.in‚ 1755.46875 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚623.16552734375( proc.memory.availableMB‚ 1870.2734375 cpu‚25.308717429120524 disk.rbd0.out‚0 disk.sda.in‚0.5625 disk.sda1.in‚0$ memory_percent‚88.58475685119629* proc.memory.percent‚16.632938385009766 network.sent‚ 2.3954865e+07% disk./.usageGB‚1108.3437385559082 disk.rbd0.in‚0( disk.nvme0c0n1.out‚67261.60595703125¢îúo2à$- _wandbruntime‚8103  _runtime‚8103_w­Éá$à _runtime‚8103.166866494 _step‚785 ' lr-AdamW/pg1‚4.7988307715153483e-05 ' lr-AdamW/pg2‚4.7988307715153483e-05 trainer/global_step‚9599 % _timestamp‚1.7884456631221666e+09»-pÎâ$È trainer/global_step‚9599 % _timestamp‚1.7884456631221666e+09 _runtime‚8103.166866494 _step‚785 ' lr-AdamW/pg1‚4.7988307715153483e-05 ' lr-AdamW/pg2‚4.7988307715153483e-05‘Ö™c%2ã$- _wandbruntime‚8103  _runtime‚8103üæ^ä$•P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17800833908224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141477411 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141477405 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1086585071359488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1068784263842816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24374661383680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91781069 trainer/global_step‚9599 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178899410 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚914314587702272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17800833908224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1068784237451264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178879376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24374612890112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17803434117632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17803434117632 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1069550757942114 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚938689172457984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1086585071359488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91781066 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1086587697960448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1086587697960448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _step‚786 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158242516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚914314559567872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.curren¦“’[½ t‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17771340547938 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141477411 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376821 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1051779417394176 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1068784237451264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1068784263842816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158242040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178899410 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚938689249085952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141477405 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1051779442822144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320356781 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1069553381654750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66461447å¾TÐ (å$šP XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91781069 trainer/global_step‚9599 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178899410 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚914314587702272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17800833908224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1068784237451264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178879376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24374612890112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17803434117632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17803434117632 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1069550757942114 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚938689172457984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1086585071359488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91781066 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1086587697960448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1086587697960448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _step‚786 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158242516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚914314559567872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17771340547938 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141477411 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320376821 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1051779417394176 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1068784237451264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1068784263842816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158242040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178899410 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚938689249085952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141477405 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1051779442822144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320356781 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1069553381654750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66461447 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17800833908224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141477411 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141477405 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1086585071359488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1068784263842816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24374661383680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376’JÈú2æ$- _wandbruntime‚8103  _runtime‚8103H}I}&'ç$ N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1069661552171898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24377117923840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚938784144257024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1069658928459054 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚787 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158257887 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17802666236928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17773169890606 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1068892203548160 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17805266459648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚9599 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178897610 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320409317 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1051885783996416 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141491647 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚914406975755776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178917670 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1086697470007808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1051885758568448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91790265 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1086697470007808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66468071 % _timestamp‚1.7884456639007578e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1068892177156608 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1086694843393536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚914406947621376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178897610 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91790262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¼êöõ¨#è$¥N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178917670 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1086697470007808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1051885758568448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91790265 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1086697470007808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66468071 % _timestamp‚1.7884456639007578e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1068892177156608 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1086694843393536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚914406947621376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178897610 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91790262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1069661552171898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24377117923840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚938784144257024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1069658928459054 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚787 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158257887 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17802666236928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17773169890606 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1068892203548160 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17805266459648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚9599 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178897610 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320409317 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1051885783996416 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141491647 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚914406975755776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 E<¸ä΃DeviceStatsMonitor.on_train_batch_end/active.large_pool.peak‚2753 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17805266459648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24377168501248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968“¯¬EA2é$- _wandbruntime‚8103  _runtime‚8103!Ìx)[ê$Õ 0loss/FRt1/assignment_loss‚0.9851768854118529 0loss/FRt3/assignment_loss‚2.7717823962370556 /loss/FBt1/assignment_loss‚0.068257884615244 %loss/total_loss‚37.04760213947998 2loss/SRqqt2/assignment_loss‚0.8333351183324479 2loss/SRqqt4/assignment_loss‚1.1094388119584204 /loss/FRt3/detection_loss‚2.7826132396545185 /loss/FBt4/detection_loss‚2.5989804416894913 trainer/global_step‚9599 2loss/SRqqt3/assignment_loss‚1.4880987243617283 1loss/SRqqt1/detection_loss‚0.7895418068310663 /loss/FBt1/detection_loss‚0.6397882843972184 /loss/FBt2/detection_loss‚3.0714977234601974 2loss/SRqqt1/assignment_loss‚0.7360581397148434 2loss/FBt3/assignment_loss‚0.002268885403671967 1loss/SRqqt2/detection_loss‚1.7141295617016463 0loss/SRqqt3/detection_loss‚2.222590700668447 1loss/SRqqt4/detection_loss‚1.6618433848899954 % _timestamp‚1.7884456639032183e+09 0loss/FRt2/assignment_loss‚2.0175746186919836 0loss/FBt4/assignment_loss‚0.6004597512073815 .loss/FRt1/detection_loss‚1.562298195569643 epoch‚7 0loss/FRt4/assignment_loss‚2.4850531981346453 0loss/FBt2/assignment_loss‚0.2465950858895667 .loss/FRt4/detection_loss‚2.177978294009858 .loss/FBt3/detection_loss‚2.138438642024994 .loss/FRt2/detection_loss‚2.343802364624065 _runtime‚8103.947882652 _step‚788ÉÄ›¦`ë$Ú 1loss/SRqqt2/detection_loss‚1.7141295617016463 0loss/SRqqt3/detection_loss‚2.222590700668447 1loss/SRqqt4/detection_loss‚1.6618433848899954 % _timestamp‚1.7884456639032183e+09 0loss/FRt2/assignment_loss‚2.0175746186919836 0loss/FBt4/assignment_loss‚0.6004597512073815 .loss/FRt1/detection_loss‚1.562298195569643 epoch‚7 0loss/FRt4/assignment_loss‚2.4850531981346453 0loss/FBt2/assignment_loss‚0.2465950858895667 .loss/FRt4/detection_loss‚2.177978294009858 .loss/FBt3/detection_loss‚2.138438642024994 .loss/FRt2/detection_loss‚2.343802364624065 _runtime‚8103.947882652 _step‚788 0loss/FRt1/assignment_loss‚0.9851768854118529 0loss/FRt3/assignment_loss‚2.7717823962370556 /loss/FBt1/assignment_loss‚0.068257884615244 %loss/total_loss‚37.04760213947998 2loss/SRqqt2/assignment_loss‚0.8333351183324479 2loss/SRqqt4/assignment_loss‚1.1094388119584204 /loss/FRt3/detection_loss‚2.7826132396545185 /loss/FBt4/detection_loss‚2.5989804416894913 trainer/global_step‚9599 2loss/SRqqt3/assignment_loss‚1.4880987243617283 1loss/SRqqt1/detection_loss‚0.7895418068310663 /loss/FBt1/detection_loss‚0.6397882843972184 /loss/FBt2/detection_loss‚3.0714977234601974 2loss/SRqqt1/assignment_loss‚0.7360581397148434 2loss/FBt3/assignment_loss‚0.002268885403671967”®ÝÍŠœì$‚(:‘ ê‡æÔ”¦«É gpu.0.gpu‚4.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚89.096' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.819200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hv¹÷4í$‚(:© ê‡æÔ„Ç‚Ì disk.dm-1.in‚0.5625 network.sent‚ 2.3960115e+07 disk.rbd1.out‚ 4.048828125" disk.dm-0.out‚67359.3037109375 cpu‚25.30826748243238 disk.rbd2.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1755.46875# memory_percent‚88.5822057723999 proc.cpu.threads‚17 network.recv‚ 2.969613e+06) disk./.usagePercent‚74.38188568104394 disk.sda.in‚0.5625 disk.sda3.in‚0.5625 disk.sda3.out‚ 293.8984375* proc.memory.percent‚16.632938385009766 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 1755.46875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0! disk.rbd2.out‚623.18115234375 disk.rbd3.in‚0! proc.memory.rssMB‚ 2725.140625$ disk./.usageGB‚1108.343849182129 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚67359.3037109375' disk.nvme0c0n1.out‚67359.3037109375 disk.dm-0.in‚ 1755.46875) proc.memory.availableMB‚ 1870.69140625 disk.rbd1.in‚0 disk.sda.out‚ 293.8984375 disk.rbd0.in‚0 disk.dm-1.out‚ 293.8984375ŒBbŸî$‚(:” ù‡æÔà—ËÉ gpu.0.gpu‚66.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚224.891' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.978199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“×c]+ï$‚(:  ù‡æÔþä¤Ë disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.048828125 disk.rbd2.in‚0 disk.dm-1.out‚ 294.640625 network.recv‚ 2.976145e+06) disk./.usagePercent‚74.38189028917955 disk.nvme0c0n1.in‚ 1755.46875) proc.memory.availableMB‚ 1870.69140625 cpu‚25.307797258971593 disk.rbd0.out‚0 disk.dm-0.out‚67359.87109375 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1755.46875 disk.nvme0n1.in‚ 1755.46875# memory_percent‚88.5822057723999 proc.cpu.threads‚17% disk./.usageGB‚1108.3439178466797% disk.nvme0c0n1.out‚67359.87109375 disk.rbd3.out‚0 disk.sda3.out‚ 294.640625 disk.rbd2.out‚ 623.330078125 disk.sda.in‚0.5625 disk.rbd3.in‚0# disk.nvme0n1.out‚67359.87109375 disk.sda3.in‚0.5625 disk.rbd1.in‚0 disk.sda.out‚ 294.640625 disk.sda1.out‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2725.140625* proc.memory.percent‚16.632938385009766 network.sent‚ 2.4069255e+07 disk.rbd0.in‚0 disk.dm-1.in‚0.5625PÀðþ2ð$- _wandbruntime‚8144  _runtime‚8144‰õ›àÉñ$à ' lr-AdamW/pg1‚4.6160858461892574e-05 ' lr-AdamW/pg2‚4.6160858461892574e-05 trainer/global_step‚9649 % _timestamp‚1.7884457043559887e+09 _runtime‚8144.400718996 _step‚789»¬Îò$È ' lr-AdamW/pg2‚4.6160858461892574e-05 trainer/global_step‚9649 % _timestamp‚1.7884457043559887e+09 _runtime‚8144.400718996 _step‚789 ' lr-AdamW/pg1‚4.6160858461892574e-05•$·i´2ó$- _wandbruntime‚8144  _runtime‚8144eþÒ(ô$–P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1074181222718464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚918933962243072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚943433041813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142189211 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66798201 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17895064966656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1092076314076672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17892464757248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1092073687475712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17895064966656 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142189205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179794428 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _step‚790 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1074181249110016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24499000845312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17862821272546 trainer/global_step‚9649 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24499051436032 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1074961921092958 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179794428 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159039070 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159038589 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1092073687475712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1074181249110016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1074959297380322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 % _timestamp‚1.7884457043567839e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚943432963088384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179794428 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _step‚790 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1074181249110016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24499000845312 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17862821272546 trainer/global_step‚9649 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24499051436032 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1074961921092958 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179794428 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159039070 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159038589 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1092073687475712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1074181249110016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1074959297380322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 % _timestamp‚1.7884457043567839e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚943432963088384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚481 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚918933962243072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚943433041813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142189211 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66798201 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17895064966656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1092076314076672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17892464757248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1092073687475712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17895064966656 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142189205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159054498 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92250062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179832722 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24501488601088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚943527919706624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1074289188815360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66804903 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1057202817282048 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17867248900090 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚943527838897664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool~½¼Ù.allocated‚1074 _step‚791 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1074289162423808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1075070091610106 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159054968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9649 _runtime‚8145.183531683 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142203447 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17894297085952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1092183459509760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322016103 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179812662 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24501541275648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142203447 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17896897308672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1075067467897262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17896897308672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1074289162423808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179832722 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944£ßwØ+'ú$¥N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1074289162423808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179832722 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚919026378430976 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1057202842710016 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17894297085952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159054498 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92250062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179832722 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24501488601088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚943527919706624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1074289188815360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66804903 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1057202817282048 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17867248900090 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚943527838897664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 _step‚791 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1074289162423808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1075070091610106 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159054968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9649 _runtime‚8145.183531683 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142203447 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17894297085952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1092183459509760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322016103 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179812662 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24501541275648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142203447 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17896897308672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1075067467897262 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17896897308672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103—êÑî2û$- _wandbruntime‚8145  _runtime‚8145œÈßÃWü$Ñ 0loss/FRt1/assignment_loss‚1.3064828835151814 1loss/SRqqt1/assignment_loss‚1.154492169545847 1loss/FBt1/assignment_loss‚0.11919452843606283 0loss/FBt3/assignment_loss‚0.3724605787429027 /loss/FRt3/detection_loss‚2.5809898835884124 /loss/FRt4/detection_loss‚2.3349575977376165 /loss/FBt4/assignment_loss‚0.358414718189225 /loss/FRt4/assignment_loss‚2.392545946100925 /loss/SRqqt4/detection_loss‚1.58232616558671 0loss/FRt2/assignment_loss‚2.2886341874663896 0loss/FRt3/assignment_loss‚3.0521039241238643 2loss/SRqqt3/assignment_loss‚1.4556243488421807 1loss/SRqqt4/assignment_loss‚1.208982899801975 /loss/FRt1/detection_loss‚1.8189269024188872 _step‚792 /loss/FBt1/detection_loss‚0.5846340015351054 .loss/FBt3/detection_loss‚2.366778884615217 .loss/FBt4/detection_loss‚2.200318170918359 0loss/FBt2/assignment_loss‚0.4831700121618009 /loss/FRt2/detection_loss‚2.5048171088502214 0loss/SRqqt2/detection_loss‚2.178763706332598 /loss/FBt2/detection_loss‚2.9932389770235335 _runtime‚8145.185258715 2loss/SRqqt2/assignment_loss‚1.2943436546394458 1loss/SRqqt1/detection_loss‚1.0469833401148207 trainer/global_step‚9649 % _timestamp‚1.7884457051402884e+09 0loss/SRqqt3/detection_loss‚2.489793511537405 &loss/total_loss‚40.168978101824685 epoch‚7›¸\ý$Ö .loss/FBt3/detection_loss‚2.366778884615217 .loss/FBt4/detection_loss‚2.200318170918359 0loss/FBt2/assignment_loss‚0.4831700121618009 /loss/FRt2/detection_loss‚2.5048171088502214 0loss/SRqqt2/detection_loss‚2.178763706332598 /loss/FBt2/detection_loss‚2.9932389770235335 _runtime‚8145.185258715 2loss/SRqqt2/assignment_loss‚1.2943436546394458 1loss/SRqqt1/detection_loss‚1.0469833401148207 trainer/global_step‚9649 % _timestamp‚1.7884457051402884e+09 0loss/SRqqt3/detection_loss‚2.489793511537405 &loss/total_loss‚40.168978101824685 epoch‚7 0loss/FRt1/assignment_loss‚1.3064828835151814 1loss/SRqqt1/assignment_loss‚1.154492169545847 1loss/FBt1/assignment_loss‚0.11919452843606283 0loss/FBt3/assignment_loss‚0.3724605787429027 /loss/FRt3/detection_loss‚2.5809898835884124 /loss/FRt4/detection_loss‚2.3349575977376165 /loss/FBt4/assignment_loss‚0.358414718189225 /loss/FRt4/assignment_loss‚2.392545946100925 /loss/SRqqt4/detection_loss‚1.58232616558671 0loss/FRt2/assignment_loss‚2.2886341874663896 0loss/FRt3/assignment_loss‚3.0521039241238643 2loss/SRqqt3/assignment_loss‚1.4556243488421807 1loss/SRqqt4/assignment_loss‚1.208982899801975 /loss/FRt1/detection_loss‚1.8189269024188872 _step‚792 /loss/FBt1/detection_loss‚0.5846340015351054˜§*IJ“þ$‚(:ˆ —ˆæÔÖÇ›É gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚91.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UVZÙ7ÿ$‚(:¬ —ˆæÔ–¸ÖË disk.rbd1.in‚0 disk.rbd2.out‚ 625.044921875 disk.nvme0c0n1.in‚ 1755.46875( disk.nvme0c0n1.out‚67457.93505859375# disk.dm-0.out‚67457.93505859375# memory_percent‚88.5838508605957 disk.rbd3.in‚0 disk.sda3.out‚ 295.73046875' proc.memory.availableMB‚ 1870.421875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda.out‚ 295.73046875 disk.sda2.out‚0 disk.nvme0n1.in‚ 1755.46875! proc.memory.rssMB‚ 2725.140625 network.sent‚ 2.4172938e+07* proc.memory.percent‚16.632938385009766 network.recv‚ 2.986948e+06& disk.nvme0n1.out‚67457.93505859375 disk.rbd0.in‚0) disk./.usagePercent‚74.38190283354874 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.in‚0.5625% disk./.usageGB‚1108.3441047668457 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.048828125 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1755.46875 cpu‚25.30685004708367 disk.sda2.in‚0 disk.dm-1.out‚ 295.73046875KI"›’€%‚(:‡ ¦ˆæÔÍí‡É gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K’Z‚8%‚(:­ ¦ˆæÔ”„Ì disk.rbd2.in‚0 disk.sda.out‚ 296.05859375 disk.nvme0n1.in‚ 1755.46875 disk.rbd0.out‚0* proc.memory.percent‚16.632938385009766 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 296.05859375' proc.memory.availableMB‚ 1870.671875 disk.rbd1.out‚ 4.048828125 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.out‚ 625.068359375 disk.sda3.in‚0.5625$ memory_percent‚88.58232498168945 network.recv‚ 2.992641e+06 disk.rbd3.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚67458.26708984375 disk.dm-1.out‚ 296.05859375 disk.sda2.out‚0 disk.dm-0.in‚ 1755.46875! proc.memory.rssMB‚ 2725.140625 cpu‚25.31262611344409 network.sent‚ 2.4270734e+07) disk./.usagePercent‚74.38186392040353& disk.nvme0n1.out‚67458.26708984375% disk./.usageGB‚1108.3435249328613 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 1755.46875# disk.dm-0.out‚67458.26708984375 proc.cpu.threads‚17…p\€‚%‚(:’ µˆæÔ•ˆÇÉ gpu.0.gpu‚63.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚36.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€MçÁ6ƒ%‚(:« µˆæÔ›ÂÚË# disk.dm-0.out‚67555.97119140625 disk.sda.in‚0.5625) proc.memory.availableMB‚ 1870.43359375 proc.cpu.threads‚17% disk./.usageGB‚1108.3435554504395 disk.sda1.out‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚67555.97119140625 disk.sda3.in‚0.5625 disk.sda.out‚ 296.6015625$ memory_percent‚88.58377933502197 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 1755.46875 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2725.140625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 296.6015625( disk.nvme0c0n1.out‚67555.97119140625 disk.dm-0.in‚ 1755.46875* proc.memory.percent‚16.632938385009766 cpu‚25.32124632517848 disk.rbd2.out‚ 625.193359375 network.recv‚ 2.997171e+06( disk./.usagePercent‚74.3818659684638 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1755.46875 disk.rbd1.out‚ 4.048828125 network.sent‚ 2.4276683e+07 disk.rbd0.in‚0 disk.dm-1.out‚ 296.6015625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0Á3Ì‚œ„%‚(:‘ ĈæÔñÒ¤É gpu.0.gpu‚6.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚34.0 gpu.0.powerWatts‚87.719' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.543799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÕ†,…%‚(:¡ ĈæÔ±¾ÅÌ disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 1755.46875' disk.nvme0c0n1.out‚67556.4013671875$ memory_percent‚88.58504295349121* proc.memory.percent‚16.632938385009766 cpu‚25.32873401214387 disk.rbd0.in‚0 disk.dm-1.out‚ 297.15625 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 297.15625 proc.cpu.threads‚17 network.recv‚ 3.001028e+06 disk.sda.out‚ 297.15625 disk.sda2.out‚0 network.sent‚ 2.4281557e+07 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.sda3.in‚0.5625" disk.dm-0.out‚67556.4013671875) disk./.usagePercent‚74.38185112002682$ disk./.usageGB‚1108.343334197998 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1755.46875% disk.nvme0n1.out‚67556.4013671875 disk.rbd0.out‚0 disk.rbd2.out‚ 625.208984375 disk.dm-0.in‚ 1755.46875( proc.memory.availableMB‚ 1870.2265625! proc.memory.rssMB‚ 2725.140625 disk.rbd1.out‚ 4.048828125ì»Äà2†%- _wandbruntime‚8214  _runtime‚8214xVcM‡%ÿT aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012864 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚67140415 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚92840947 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 143050440 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 159981360 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 159981492 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚130 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚924096050975232 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚24580012765696 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2649 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2711634944 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _runtime‚8214.590090261 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 180571494 ZLDeviceStaøúo©8tsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 143050445 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2649 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 136819200 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23672651776 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 143050445 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 180587468 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚924096033326592 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9671 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1080278033140224 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 180571494 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1098238495975936 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637913 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621934 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119170560 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1575 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚17958512117248 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23672651776 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 _step‚793 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3303014400 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 180587468 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1080278008845824 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚17928726788518 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23672651776 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621934 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚17960462835712† T/Š*ˆ%„U _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1080278008845824 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚17928726788518 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23672651776 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323621934 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚17960462835712 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 1975012864 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚67140415 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚92840947 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 143050440 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 159981360 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 159981492 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚130 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚924096050975232 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚24580012765696 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚2649 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 2711634944 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _runtime‚8214.590090261 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 180571494 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 143050445 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2649 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1074 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 20788341248 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 136819200 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23672651776 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 143050445 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 180587468 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚924096033326592 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9671 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1080278033140224 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 180571494 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1098238495975936 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637913 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 323621934 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119170560 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1575 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚17958512117248 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23672651776 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 _step‚793 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 3303014400 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 180587468™Ù)I2‰%- _wandbruntime‚8215  _runtime‚8215ô|[)Š%‰S \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1575 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚17960937038848 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 159993149 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚948738639469056 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1098310753754112 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23672651776 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 143061339 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚1080971579727590 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2649 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚924158033329664 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚24580486968832 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 trainer/global_step‚9671 _runtime‚8215.559613325 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚67144405 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚948738502649856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233žœèeZ‹%ŽS KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1575 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚17960937038848 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 159993149 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚948738639469056 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1098310753754112 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23672651776 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 143061339 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚1080971579727590 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2649 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚924158033329664 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚24580486968832 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 trainer/global_step‚9671 _runtime‚8215.559613325 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚67144405 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚948738502649856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376š;z圌%‚(:‘ ÓˆæÔ·•‚É gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.97' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.19400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶aË8%‚(:­ ÓˆæÔ¹þˆË disk.rbd1.out‚ 4.048828125 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚67654.38720703125! proc.memory.rssMB‚ 2725.140625 network.recv‚ 3.005382e+06 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 297.6015625 disk.dm-1.in‚0.5625* proc.memory.percent‚16.632938385009766 network.sent‚ 2.428779e+07 disk.sda2.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚67654.38720703125 cpu‚25.336239376844766 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.nvme0n1.in‚ 1755.46875 disk.dm-0.in‚ 1755.46875$ memory_percent‚88.58447074890137% disk./.usageGB‚1108.3434143066406! disk.rbd2.out‚625.31103515625 disk.sda.out‚ 297.6015625( disk.nvme0c0n1.out‚67654.38720703125 disk.dm-1.out‚ 297.6015625 proc.cpu.threads‚17) disk./.usagePercent‚74.38185649618504 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 1755.46875( proc.memory.availableMB‚ 1870.3203125Â}ðÙ2Ž%- _wandbruntime‚8229  _runtime‚8229öoœAm%£§ 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4768678545951843 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1350403130054474 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12860064208507538 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14056578278541565 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 jet/accuracy_6_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4725707173347473 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4963342547416687 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5604474544525146 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1694723665714264 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027188941836357117 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03993048518896103 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889692664146423 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5996671915054321 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.003510494949296117 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5120653510093689 particle/accuracy_1_of_1‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06178160756826401 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5213836431503296 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.985663115978241 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.13808885216712952 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22200392186641693 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7088330388069153 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21913579106330872 jet/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5371584892272949 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015986140817403793 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5113754272460938 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3342593014240265 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9762800335884094 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013739841058850288 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12888890504837036 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4566449522972107 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5067170858383179 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5476593971252441 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49580031633377075 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.190221905708313 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2912421226501465 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4441177546977997 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9928166270256042 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5208112597465515 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.371090292930603 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6238843202590942 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.007258064113557339 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 *jet/accuracy_1_of_3‚0.8730136156082153 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970253705978394 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4623781144618988 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5659722685813904 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5105816721916199 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05182095989584923 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4182250499725342 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5454535484313965 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5982142686843872 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915789365768433 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5359190702438354 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3749999701976776 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933907985687256 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5071581602096558 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 trainer/global_step‚9671 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46810343861579895 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.21128657460212708 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +particle/specificity‚0.9512601494789124 particle/accuracy_6_of_7‚1 _runtime‚ 8229.47594076 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5824437141418457 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12117522209882736 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08616741746664047 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14463600516319275 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 &particle/f_score‚0.396714448928833 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5207504630088806 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28303906321525574 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018115395680069923 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3191198408603668 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_5_of_7‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17783397436141968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1911533921957016 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41383931040763855 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4517543613910675 particle/accuracy_3_of_6‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5807181596755981 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 epoch‚7 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31374138593673706 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35798609256744385 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1911533921957016 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0028737422544509172 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36111000180244446 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4234495759010315 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4879879951477051 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9524705410003662 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5747562646865845 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876149296760559 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1486567258834839 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1882321834564209 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14506171643733978 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9977324604988098 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.407184362411499 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5822916626930237 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6398809552192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8225924372673035 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49828892946243286 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.09615384787321091 particle/accuracy_4_of_7‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36683860421180725 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5184478759765625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9925925731658936 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1315789520740509 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.15833333134651184 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4836806058883667 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4030366837978363 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4956720173358917 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 particle/accuracy_2_of_5‚1 -"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5625 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9976640343666077 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0634920671582222 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3221372365951538 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3657407760620117 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13419541716575623 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚40.31291138314419 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957354068756104 *jet/accuracy_1_of_4‚0.9209160804748535 particle/accuracy_3_of_7‚1 particle/accuracy_7_of_7‚0.5 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07372277975082397 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5010519623756409 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25828060507774353 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993831992149353 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.012026369571685791 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22287842631340027 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15186072885990143 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0760558620095253 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5032907128334045 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5283835530281067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02070402354001999 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3981480896472931 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47685185074806213 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 *jet/accuracy_2_of_6‚0.5512820482254028 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.280081182718277 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.513748824596405 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5555051565170288 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18434344232082367 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 (particle/accuracy‚0.8515521287918091 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 _step‚795 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0486268624663353 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44168582558631897 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19165270030498505 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2105855941772461 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_1_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008225012570619583 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16064134240150452 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6274682283401489 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11794289946556091 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11442895978689194 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 5 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32663694024086 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4949495196342468 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.004926473367959261 particle/accuracy_1_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5078775882720947 *jet/accuracy_2_of_3‚0.2607910633087158 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.015359476208686829 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029026975855231285 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5684976577758789 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5071581602096558 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 % _timestamp‚1.7884457894283826e+09 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21780423820018768 particle/accuracy_3_of_5‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4261529743671417 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860629439353943 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13948789238929749 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.650678813457489 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5796770453453064 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902175068855286 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06772400438785553 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977408051490784 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.47378823161125183 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20945873856544495 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4298936724662781 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5185946226119995 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26621338725090027 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01924288459122181 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5471392273902893 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5479%^ZÝ &785203933716 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.366015762090683 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46613383293151855 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.011437909677624702 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22365351021289825 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3137931227684021 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.544205367565155 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5415753126144409 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17018024623394012 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4466666877269745 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04591194912791252 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_5_of_7‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11775287985801697 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.03240324556827545 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880130290985107 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15843743085861206 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 jet/accuracy_6_of_7‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1976664811372757 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13931936025619507 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.09375 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10880468785762787 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6045117974281311 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008724340237677097 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9701867699623108 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_1‚0.4902116358280182 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15014153718948364 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14636804163455963 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48220041394233704 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9901229739189148 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16227643191814423 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.20384463667869568 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5047460198402405 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2823048532009125 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22503386437892914 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21936406195163727 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48949673771858215 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03129708021879196 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29918617010116577 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3680555522441864 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.450617253780365 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9922339916229248 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26172399520874023 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5128670930862427 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11840827763080597 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 jet/accuracy_2_of_7‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18812479078769684 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.12619046866893768 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927570223808289 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5589743852615356 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8333333730697632 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7494555711746216 +jet/accuracy_2_of_4‚0.39049026370048523 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42736369371414185 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5891751050949097 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18996714055538177 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +particle/sensitivity‚0.3128455579280853 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 6ÜåY%¨§ @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3680555522441864 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.015359476208686829 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029026975855231285 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5684976577758789 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5071581602096558 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 % _timestamp‚1.7884457894283826e+09 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21780423820018768 particle/accuracy_3_of_5‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4261529743671417 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860629439353943 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13948789238929749 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.650678813457489 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5796770453453064 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902175068855286 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06772400438785553 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977408051490784 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.47378823161125183 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20945873856544495 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4298936724662781 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5185946226119995 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26621338725090027 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01924288459122181 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5471392273902893 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5479785203933716 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.366015762090683 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46613383293151855 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.011437909677624702 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22365351021289825 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3137931227684021 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.544205367565155 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5415753126144409 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17018024623394012 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4466666877269745 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04591194912791252 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 particle/accuracy_5_of_7‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.056737590581178665 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11775287985801697 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.03240324556827545 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880130290985107 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15843743085861206 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 jet/accuracy_6_of_7‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1976664811372757 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13931936025619507 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.09375 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10880468785762787 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6045117974281311 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008724340237677097 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9701867699623108 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_1‚0.4902116358280182 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15014153718948364 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14636804163455963 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48220041394233704 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9901229739189148 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16227643191814423 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.20384463667869568 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5047460198402405 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2823048532009125 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22503386437892914 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21936406195163727 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48949673771858215 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03129708021879196 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29918617010116577 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3680555522441864 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.450617253780365 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9922339916229248 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26172399520874023 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5128670930862427 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11840827763080597 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 jet/accuracy_2_of_7‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18812479078769684 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.12619046866893768 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927570223808289 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5589743852615356 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8333333730697632 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7494555711746216 +jet/accuracy_2_of_4‚0.39049026370048523 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42736369371414185 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5891751050949097 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18996714055538177 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +particle/sensitivity‚0.3128455579280853 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4768678545951843 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1350403130054474 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12860064208507538 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14056578278541565 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 jet/accuracy_6_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4725707173347473 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4963342547416687 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5604474544525146 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1694723665714264 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027188941836357117 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03993048518896103 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889692664146423 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5996671915054321 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.003510494949296117 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5120653510093689 particle/accuracy_1_of_1‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06178160756826401 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5213836431503296 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.985663115978241 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.13808885216712952 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22200392186641693 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7088330388069153 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21913579106330872 jet/accuracy_1_of_6‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5371584892272949 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015986140817403793 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5113754272460938 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3342593014240265 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9762800335884094 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013739841058850288 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12888890504837036 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4566449522972107 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5067170858383179 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5476593971252441 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49580031633377075 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.190221905708313 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2912421226501465 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4441177546977997 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9928166270256042 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5208112597465515 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.371090292930603 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6238843202590942 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.007258064113557339 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 *jet/accuracy_1_of_3‚0.8730136156082153 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970253705978394 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4623781144618988 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5659722685813904 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5105816721916199 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05182095989584923 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4182250499725342 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5454535484313965 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5982142686843872 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9915789365768433 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5359190702438354 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3749999701976776 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933907985687256 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5071581602096558 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 trainer/global_step‚9671 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05¿}Ú›Ê9 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46810343861579895 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.21128657460212708 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +particle/specificity‚0.9512601494789124 particle/accuracy_6_of_7‚1 _runtime‚ 8229.47594076 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5824437141418457 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12117522209882736 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08616741746664047 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14463600516319275 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 &particle/f_score‚0.396714448928833 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5207504630088806 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28303906321525574 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018115395680069923 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3191198408603668 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.01937984675168991 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_5_of_7‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17783397436141968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1911533921957016 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41383931040763855 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4517543613910675 particle/accuracy_3_of_6‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5807181596755981 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 epoch‚7 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31374138593673706 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35798609256744385 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1911533921957016 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0028737422544509172 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36111000180244446 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4234495759010315 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4879879951477051 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9524705410003662 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5747562646865845 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876149296760559 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1486567258834839 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1882321834564209 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14506171643733978 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9977324604988098 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.407184362411499 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5822916626930237 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6398809552192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8225924372673035 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.49828892946243286 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.09615384787321091 particle/accuracy_4_of_7‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36683860421180725 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5184478759765625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9925925731658936 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1315789520740509 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.15833333134651184 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4836806058883667 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4030366837978363 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4956720173358917 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 particle/accuracy_2_of_5‚1 -"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5625 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9976640343666077 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0634920671582222 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3221372365951538 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3657407760620117 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13419541716575623 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚40.31291138314419 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957354068756104 *jet/accuracy_1_of_4‚0.9209160804748535 particle/accuracy_3_of_7‚1 particle/accuracy_7_of_7‚0.5 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07372277975082397 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5010519623756409 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25828060507774353 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993831992149353 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.012026369571685791 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22287842631340027 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15186072885990143 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0760558620095253 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5032907128334045 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5283835530281067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02070402354001999 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3981480896472931 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47685185074806213 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 *jet/accuracy_2_of_6‚0.5512820482254028 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.280081182718277 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.513748824596405 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5555051565170288 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18434344232082367 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 (particle/accuracy‚0.8515521287918091 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 _step‚795 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0486268624663353 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44168582558631897 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19165270030498505 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2105855941772461 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_1_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008225012570619583 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16064134240150452 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6274682283401489 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11794289946556091 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11442895978689194 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 5 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32663694024086 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4949495196342468 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.004926473367959261 particle/accuracy_1_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5078775882720947 *jet/accuracy_2_of_3‚0.2607910633087158 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0›á!ãè‘%‚(:„ âˆæÔך³É gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚30.0 gpu.0.powerWatts‚76.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.262 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø<ã0’%‚(:¥ âˆæÔŠí¬Ë disk.sda1.in‚0 disk.nvme0n1.in‚ 1755.46875 disk.rbd0.in‚0% disk./.usageGB‚1112.1689949035645 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.sda.out‚ 298.359375! proc.memory.rssMB‚ 4594.578125 cpu‚25.341013314546778 proc.cpu.threads‚17 disk.rbd2.in‚0& proc.memory.availableMB‚ 4.00390625% disk.nvme0n1.out‚68381.6337890625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚68381.6337890625 disk.dm-0.in‚ 1755.46875 disk.dm-1.out‚ 298.359375! disk.rbd2.out‚625.32666015625 disk.sda.in‚0.5625$ memory_percent‚99.97556209564209 disk.rbd1.in‚0 disk.rbd1.out‚ 4.048828125 network.sent‚ 2.439584e+07 network.recv‚ 3.011658e+06 disk.rbd3.in‚0 disk.sda3.out‚ 298.359375 disk.nvme0c0n1.in‚ 1755.46875" disk.dm-0.out‚68378.8837890625) disk./.usagePercent‚74.63859441992051 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.sda1.out‚0 disk.drbd1777.out‚0* proc.memory.percent‚28.043079376220703ƒËð““%‚(:ˆ ñˆæÔíضÉ gpu.0.gpu‚54.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.079 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*8n8”%‚(:­ ñˆæÔÇÆ„Ë% disk./.usageGB‚1108.5179557800293% disk.nvme0n1.out‚73179.5751953125 disk.dm-0.in‚ 1755.46875 disk.dm-1.in‚0.5625! disk.rbd2.out‚626.90087890625 disk.sda.out‚ 298.92578125) proc.memory.availableMB‚ 3746.12890625 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2725.609375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1755.46875 disk.nvme0c0n1.in‚ 1755.46875" disk.dm-0.out‚73179.5751953125 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 2.4520356e+07( disk./.usagePercent‚74.3935701209142 disk.sda3.out‚ 298.92578125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0$ memory_percent‚77.13544368743896 disk.rbd3.in‚0' disk.nvme0c0n1.out‚73179.5751953125 disk.dm-1.out‚ 298.92578125 disk.drbd1777.in‚0 disk.rbd2.in‚0 network.recv‚ 3.017983e+06 disk.rbd1.out‚ 4.048828125* proc.memory.percent‚16.635799407958984 cpu‚25.34042749784214š8y2•%- _wandbruntime‚8262  _runtime‚8262ÿ÷+Æ–%À & lr-AdamW/pg2‚4.436416803310605e-05 trainer/global_step‚9699 % _timestamp‚1.7884458227321754e+09 _runtime‚ 8262.77682747 _step‚796 & lr-AdamW/pg1‚4.436416803310605e-05`yõË—%Å & lr-AdamW/pg2‚4.436416803310605e-05 trainer/global_step‚9699 % _timestamp‚1.7884458227321754e+09 _runtime‚ 8262.77682747 _step‚796 & lr-AdamW/pg1‚4.436416803310605e-05œÌH2˜%- _wandbruntime‚8262  _runtime‚8262(^UU(™%–P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17986696600144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18013987394560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181185933 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93200249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18016587603968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884458227329285e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1084224994921472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1084225021313024 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1102241608916992 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1084224994921472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚952113934189568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1066823629599744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160576178 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1084810326199888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143587327 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚952114002428928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1084807702487252 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18016587603968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67375926 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚797 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143587333 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1102238982316032 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1102241608916992 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1102238982316032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181165899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143587333 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8262.777798869 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143587327 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181165899 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160575761 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93200252 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚68239360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚40104960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17984098315476 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24655665383424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚417 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67375512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93200249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18016587603968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884458227329285e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1084224994921472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1084225021313024 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1102241608916992 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1084224994921472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚952113934189568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1066823629599744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160576178 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1084810326199888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143587327 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚952114002428928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1084807702487252 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18016587603968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67375926 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚797 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143587333 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2759GàJÓ3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1102238982316032 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1102241608916992 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1102238982316032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181165899 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143587333 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8262.777798869 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143587327 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181165899 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160575761 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93200252 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚68239360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚40104960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17984098315476 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24655665383424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚417 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67375512 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17986696600144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18013987394560¼°±_2›%- _wandbruntime‚8263  _runtime‚8263å–)‡&'œ% N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 % _timestamp‚1.7884458235111067e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93209445 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1084915873004192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143601563 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚927550656859648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1084332961018368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24658259476992 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1102348754350080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚68226048 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18018419945984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚40091648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17988525943020 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1084332934626816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160591941 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚952208876244992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93209448 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1084332934626816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚396 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚8263.556515946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67382493 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1066929970774016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143601569 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160591542 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 trainer/global_step‚9699 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18015819723264 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181204193 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181184133 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚952208944471040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1084918496717036 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1084332934626816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160591941 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚952208876244992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93209448 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1084332934626816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚396 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚8263.556515946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67382493 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1066929970774016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143601569 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160591542 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 trainer/global_step‚9699 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18015819723264 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181204193 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181184133 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚952208944471040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1084918496717036 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143601563 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚927550656859648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400ž )€2ž%- _wandbruntime‚8263  _runtime‚8263”­àp[Ÿ%Õ 0loss/FBt2/assignment_loss‚0.1922314108038942 /loss/FBt3/assignment_loss‚0.194199878243934 /loss/FRt2/detection_loss‚2.4238835076491037 _step‚799 /loss/FRt3/detection_loss‚2.5163592227256615 /loss/FRt4/detection_loss‚2.2953856037213254 0loss/FBt1/detection_loss‚0.44555421668208306 /loss/FBt3/detection_loss‚2.1956692569785647 _runtime‚8263.558045106 2loss/SRqqt2/assignment_loss‚1.0676385856213721 .loss/FBt4/detection_loss‚2.707650532325109 0loss/FRt1/assignment_loss‚1.1700933413072065 2loss/SRqqt4/assignment_loss‚1.4634882914159741 1loss/FBt1/assignment_loss‚0.14474219919294085 1loss/SRqqt4/detection_loss‚1.4656558174051737 2loss/SRqqt1/assignment_loss‚0.8648391865247574 1loss/FBt4/assignment_loss‚0.33188376082883525 .loss/FBt2/detection_loss‚3.270804742972056 epoch‚8 0loss/FRt2/assignment_loss‚2.4940799568213667 /loss/FRt3/assignment_loss‚3.822768460620533 2loss/SRqqt3/assignment_loss‚1.2476489133135016 .loss/FRt1/detection_loss‚1.371395471167158 % _timestamp‚1.7884458235133998e+09 /loss/FRt4/assignment_loss‚2.614228150019279 0loss/SRqqt3/detection_loss‚2.559308577861105 1loss/SRqqt1/detection_loss‚0.8418000362908993 1loss/SRqqt2/detection_loss‚1.8182809571636485 %loss/total_loss‚39.51959007765548 trainer/global_step‚9699TúW_` %Ú 1loss/SRqqt2/detection_loss‚1.8182809571636485 %loss/total_loss‚39.51959007765548 trainer/global_step‚9699 0loss/FBt2/assignment_loss‚0.1922314108038942 /loss/FBt3/assignment_loss‚0.194199878243934 /loss/FRt2/detection_loss‚2.4238835076491037 _step‚799 /loss/FRt3/detection_loss‚2.5163592227256615 /loss/FRt4/detection_loss‚2.2953856037213254 0loss/FBt1/detection_loss‚0.44555421668208306 /loss/FBt3/detection_loss‚2.1956692569785647 _runtime‚8263.558045106 2loss/SRqqt2/assignment_loss‚1.0676385856213721 .loss/FBt4/detection_loss‚2.707650532325109 0loss/FRt1/assignment_loss‚1.1700933413072065 2loss/SRqqt4/assignment_loss‚1.4634882914159741 1loss/FBt1/assignment_loss‚0.14474219919294085 1loss/SRqqt4/detection_loss‚1.4656558174051737 2loss/SRqqt1/assignment_loss‚0.8648391865247574 1loss/FBt4/assignment_loss‚0.33188376082883525 .loss/FBt2/detection_loss‚3.270804742972056 epoch‚8 0loss/FRt2/assignment_loss‚2.4940799568213667 /loss/FRt3/assignment_loss‚3.822768460620533 2loss/SRqqt3/assignment_loss‚1.2476489133135016 .loss/FRt1/detection_loss‚1.371395471167158 % _timestamp‚1.7884458235133998e+09 /loss/FRt4/assignment_loss‚2.614228150019279 0loss/SRqqt3/detection_loss‚2.559308577861105 1loss/SRqqt1/detection_loss‚0.8418000362908993Ÿw(¥ “¡%‚(:ˆ €‰æÔ«ùªÉ gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.427 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*´G/¢%‚(:¤ €‰æÔ—á¡Ë disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.5625 network.recv‚ 3.022452e+06% disk./.usageGB‚1108.3339767456055 disk.nvme0n1.in‚1757.625$ memory_percent‚77.13541984558105 disk.dm-0.in‚1757.625 disk.dm-1.out‚ 299.796875 disk.rbd1.in‚0 disk.rbd1.out‚ 4.048828125 disk.sda.in‚0.5625 disk.sda.out‚ 299.796875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0* proc.memory.percent‚16.635799407958984 cpu‚25.33993080920321 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.rbd2.out‚626.91650390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚73182.08447265625 disk.sda3.out‚ 299.796875 disk.dm-1.in‚0.5625 disk.nvme0c0n1.in‚1757.625( proc.memory.availableMB‚ 3746.1328125! proc.memory.rssMB‚ 2725.609375 network.sent‚ 2.4526312e+07) disk./.usagePercent‚74.38122313354533# disk.dm-0.out‚73182.08447265625 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚73182.08447265625ò¨ªž£%‚(:“ ‰æÔƒ²ÇÉ gpu.0.gpu‚82.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.686' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.13719999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}<éÛ9¤%‚(:® ‰æÔœ‰Ë disk.sda2.out‚0 disk.sda3.in‚0.5625 disk.rbd1.out‚ 4.048828125" disk.dm-0.out‚73186.1181640625 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0! disk.rbd2.out‚627.04150390625 disk.sda1.in‚0' disk.nvme0c0n1.out‚73186.1181640625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 300.5859375$ memory_percent‚77.13723182678223( proc.memory.availableMB‚ 3745.8359375 disk.dm-0.in‚ 1758.390625 disk.rbd3.out‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2725.609375! disk.nvme0c0n1.in‚ 1758.390625 disk.rbd2.in‚0 network.recv‚ 3.029302e+06 disk.rbd1.in‚0) disk./.usagePercent‚74.38122159750013 disk.dm-1.out‚ 300.5859375 disk.nvme0n1.in‚ 1758.390625 disk.rbd0.in‚0 cpu‚25.339378433385562 network.sent‚ 2.4635564e+07% disk./.usageGB‚1108.3339538574219 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda.out‚ 300.5859375% disk.nvme0n1.out‚73186.1181640625* proc.memory.percent‚16.635799407958984 proc.cpu.threads‚17z¥Ð‚”¥%‚(:‰ ž‰æÔÕ¥³É gpu.0.gpu‚81.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚239.187' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Ò¬5¦%‚(:¤ ž‰æÔªøåË' proc.memory.availableMB‚ 3745.859375! proc.memory.rssMB‚ 2725.609375* proc.memory.percent‚16.635799407958984 disk.sda3.in‚0.5625$ disk.nvme0c0n1.out‚ 73284.0078125 disk.rbd2.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd177˜x^ú7.out‚0 disk.rbd1.out‚ 4.048828125 disk.sda.out‚ 301.078125 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.4640804e+07 disk.dm-1.in‚0.5625 disk.rbd3.out‚0 cpu‚25.338882258078673 disk.sda3.out‚ 301.078125" disk.nvme0c0n1.in‚ 1759.1484375 disk.dm-0.in‚ 1759.1484375" disk.nvme0n1.out‚ 73284.0078125 disk.dm-0.out‚ 73284.0078125 disk.dm-1.out‚ 301.078125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1759.1484375$ memory_percent‚77.13708877563477) disk./.usagePercent‚74.38125411045698! disk.rbd2.out‚627.05322265625 disk.sda.in‚0.5625 network.recv‚ 3.033102e+06% disk./.usageGB‚1108.3344383239746 disk.rbd0.out‚0 disk.rbd3.in‚0l–¹2§%- _wandbruntime‚8304  _runtime‚8304Žß)Ȩ% ' lr-AdamW/pg1‚4.2598610856386444e-05 ' lr-AdamW/pg2‚4.2598610856386444e-05 trainer/global_step‚9749 $ _timestamp‚1.788445864424883e+09 _runtime‚8304.469644313 _step‚800Â:‘Í©%Ç _step‚800 ' lr-AdamW/pg1‚4.2598610856386444e-05 ' lr-AdamW/pg2‚4.2598610856386444e-05 trainer/global_step‚9749 $ _timestamp‚1.788445864424883e+09 _runtime‚8304.469644313 îbõk2ª%- _wandbruntime‚8304  _runtime‚8304Eݺ(«%•P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚932077671481344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326400118 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1089622006580224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161373373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67712863 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18105618243584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1089621980188672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144299133 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1107727598432256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1107730225033216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93660049 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚956860150716928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182100985 trainer/global_step‚9749 _step‚801 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1089622006580224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚932077699615744 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072140688313344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8304.470749725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182080951 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1072140662885376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1090218865638096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1107727598432256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144299127 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18075579040084 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144299127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18105618243584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1090216241925460 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JË (¬%šP aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1089622006580224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚932077699615744 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072140688313344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8304.470749725 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182080951 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1072140662885376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1090218865638096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1107727598432256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144299127 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18075579040084 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144299127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18105618243584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1090216241925460 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326400118 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1089622006580224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161373373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67712863 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18105618243584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1089621980188672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144299133 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1107727598432256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1107730225033216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93660049 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚956860150716928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182100985 trainer/global_step‚9749 _step‚801¡Ũÿc2­%- _wandbruntime‚8305  _runtime‚8305±}¤&'®% N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161388801 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18110050795008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚956955145607680 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1090327036155244 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1107839997080576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1089729946285568 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18080006667628 % _timestamp‚1.7884458652116222e+09 _step‚802 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144313369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1090324412442400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18110050795008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326412548 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144313363 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93669245 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1089729946285568 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1089729919894016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182119245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144313363 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93669248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182099185 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0PÇM4¯%¥N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18077408382752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1089729946285568 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1089729919894016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182119245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144313363 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93669248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182099185 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161388801 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18110050795008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚956955145607680 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1090327036155244 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1107839997080576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1089729946285568 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18080006667628 % _timestamp‚1.7884458652116222e+09 _step‚802 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144313369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1090324412442400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18110050795008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326412548 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144313363 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93669245 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1095624781363668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1113216214548480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68049726 trainer/global_step‚9799 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1095018991847424 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94119849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162170033 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24909114778112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1095018991847424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026970 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚961606144635392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24909070478848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1113216214548480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚936697074156544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚805 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18199849302016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18197249092608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145010933 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚961606217069056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1095018965455872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94119852 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚936697102290944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18169658049360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1095018965455872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183016037 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 # _timestamp‚1.78844590664813e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68050181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚961606144635392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24909070478848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1113216214548480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚936697074156544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚805 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18199849302016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18197249092608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145010933 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚961606217069056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1095018965455872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94119852 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚936697102290944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18169658049360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1095018965455872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183016037 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 # _timestamp‚1.78844590664813e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68050181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233¥ÀúÔl2¿%- _wandbruntime‚8347  _runtime‚8347›û†»&'À% N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1113328613196800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18199081421312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18168889107360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162185539 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1095126905161216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095732951880608 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚936789462210048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1077564062773248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145025169 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059466 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183034297 trainer/global_step‚9799 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328039400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183014237 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162185959 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24911613623808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24911657909760 _runtime‚8347.485641852 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1095126931552768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚806 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145025163 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884459074402883e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚936789490344448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400vÑŠ +'Á%¥N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24911613623808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24911657909760 _runtime‚8347.485641852 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1095126931552768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚806 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145025163 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884459074402883e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚936789490344448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1113328613196800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18199081421312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18168889107360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162185539 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1095126905161216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095732951880608 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚936789462210048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1077564062773248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145025169 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328059466 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183034297 trainer/global_step‚9799 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328039400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183014237 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162185959¦³Eo2Â%- _wandbruntime‚8347  _runtime‚8347}·ÝüYÃ%Ó 1loss/FBt3/assignment_loss‚0.25571876037865876 1loss/FBt4/assignment_loss‚0.24486358205922443 /loss/FRt4/detection_loss‚2.2951033857058394 trainer/global_step‚9799 /loss/FRt2/detection_loss‚2.4657380308793937 1loss/SRqqt2/detection_loss‚1.7677744215696443 0loss/SRqqt3/detection_loss‚2.155103359785345 _step‚807 0loss/FRt3/assignment_loss‚2.7971971849600474 1loss/SRqqt2/assignment_loss‚0.903247133353535 2loss/SRqqt4/assignment_loss‚1.1084523102555137 .loss/FBt2/detection_loss‚2.321761332452297 2loss/SRqqt3/assignment_loss‚1.4254603686018124 $loss/total_loss‚37.1122856600235 /loss/FRt1/detection_loss‚1.4950073707150295 .loss/FBt1/detection_loss‚0.506618107103252 .loss/FBt3/detection_loss‚2.770842361450195 % _timestamp‚1.7884459074423769e+09 0loss/FRt1/assignment_loss‚1.2672992472847302 /loss/FRt4/assignment_loss‚2.283239636908878 1loss/FBt1/assignment_loss‚0.16488237022251975 1loss/FBt2/assignment_loss‚0.23619830514508067 1loss/SRqqt4/detection_loss‚1.6076454614217466 .loss/FBt4/detection_loss‚2.719274841248989 _runtime‚8347.487029787 2loss/SRqqt1/assignment_loss‚0.8101545726282947 .loss/FRt3/detection_loss‚2.763018451272258 epoch‚8 1loss/SRqqt1/detection_loss‚0.9922319030498757 /loss/FRt2/assignment_loss‚1.755453161571337•Hx&^Ä%Ø .loss/FBt3/detection_loss‚2.770842361450195 % _timestamp‚1.7884459074423769e+09 0loss/FRt1/assignment_loss‚1.2672992472847302 /loss/FRt4/assignment_loss‚2.283239636908878 1loss/FBt1/assignment_loss‚0.16488237022251975 1loss/FBt2/assignment_loss‚0.23619830514508067 1loss/SRqqt4/detection_loss‚1.6076454614217466 .loss/FBt4/detection_loss‚2.719274841248989 _runtime‚8347.487029787 2loss/SRqqt1/assignment_loss‚0.8101545726282947 .loss/FRt3/detection_loss‚2.763018451272258 epoch‚8 1loss/SRqqt1/detection_loss‚0.9922319030498757 /loss/FRt2/assignment_loss‚1.755453161571337 1loss/FBt3/assignment_loss‚0.25571876037865876 1loss/FBt4/assignment_loss‚0.24486358205922443 /loss/FRt4/detection_loss‚2.2951033857058394 trainer/global_step‚9799 /loss/FRt2/detection_loss‚2.4657380308793937 1loss/SRqqt2/detection_loss‚1.7677744215696443 0loss/SRqqt3/detection_loss‚2.155103359785345 _step‚807 0loss/FRt3/assignment_loss‚2.7971971849600474 1loss/SRqqt2/assignment_loss‚0.903247133353535 2loss/SRqqt4/assignment_loss‚1.1084523102555137 .loss/FBt2/detection_loss‚2.321761332452297 2loss/SRqqt3/assignment_loss‚1.4254603686018124 $loss/total_loss‚37.1122856600235 /loss/FRt1/detection_loss‚1.4950073707150295 .loss/FBt1/detection_loss‚0.506618107103252§æ|ÝW“Å%‚(:ˆ Ú‰æÔò¢ÄÉ gpu.0.gpu‚65.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚260.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ä7Æ%‚(:¬ Ú‰æÔî¹ßË disk.nvme0c0n1.in‚ 1760.1875 disk.dm-1.in‚0.5625 proc.cpu.threads‚17) disk./.usagePercent‚74.38142230740696 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.dm-1.out‚ 303.16796875 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.2294921875$ memory_percent‚77.13706493377686% disk./.usageGB‚1108.3369445800781 disk.nvme0n1.in‚ 1760.1875 disk.sda3.in‚0.5625) proc.memory.availableMB‚ 3745.86328125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0! disk.rbd2.out‚628.81982421875 disk.rbd0.in‚0" disk.dm-0.out‚73384.5751953125 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2725.609375* proc.memory.percent‚16.635799407958984 network.recv‚ 3.052138e+06% disk.nvme0n1.out‚73384.5751953125 disk.sda3.out‚ 303.16796875 disk.dm-0.in‚ 1760.1875 disk.rbd2.in‚0 cpu‚25.33687433600265 network.sent‚ 2.4767138e+07 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 303.16796875 disk.sda1.in‚0' disk.nvme0c0n1.out‚73384.5751953125éÑ6øžÇ%‚(:“ é‰æÔƲªÉ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚140.42' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.083999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A4«4È%‚(:© é‰æÔÆ”ÄË disk.rbd0.in‚0) disk./.usagePercent‚74.38142307542957 disk.sda.out‚ 303.96484375 disk.sda1.out‚0 network.recv‚ 3.058651e+06( disk.nvme0c0n1.out‚73482.76611328125# disk.dm-0.out‚73482.76611328125 disk.dm-1.out‚ 303.96484375 disk.drbd1777.out‚0 disk.sda.in‚0.5625 disk.sda3.out‚ 303.96484375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.rbd3.out‚0 disk.rbd2.out‚ 628.9140625# memory_percent‚77.1369457244873 disk.sda2.out‚0 disk.nvme0n1.in‚ 1760.1875 disk.dm-0.in‚ 1760.1875 disk.drbd1777.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 3745.8828125! proc.memory.rssMB‚ 2725.609375* proc.memory.percent‚16.635799407958984& disk.nvme0n1.out‚73482.76611328125 disk.nvme0c0n1.in‚ 1760.1875 disk.sda1.in‚0 cpu‚25.336393000606808 network.sent‚ 2.4876115e+07# disk./.usageGB‚1108.33695602417 disk.rbd1.in‚0 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 4.29248046875 proc.cpu.threads‚17OŸÉ%‚(:” ø‰æÔë¹ÓÉ gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚247.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.519200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼øž2Ê%‚(:§ ø‰æÔ› Ë disk.sda2.in‚0( disk.nvme0c0n1.out‚73483.10986328125 disk.drbd1777.in‚0 network.sent‚ 2.4881906e+07 disk.sda.out‚ 304.34765625 disk.rbd0.out‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2725.609375 disk.sda2.out‚0# disk.dm-0.out‚73483.10986328125$ memory_percent‚77.13699340820312* proc.memory.percent‚16.635799407958984 disk.nvme0n1.in‚ 1760.1875& disk.nvme0n1.out‚73483.10986328125 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 304.34765625$ proc.memory.availableMB‚3745.875 disk.sda1.out‚0 disk.sda3.in‚0.5625 proc.cpu.threads‚17 disk.sda3.out‚ 304.34765625 disk.dm-0.in‚ 1760.1875 network.recv‚ 3.063225e+06) disk./.usagePercent‚74.38141923531656$ disk./.usageGB‚1108.336898803711 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 1760.1875 disk.rbd1.out‚ 4.29248046875 disk.rbd2.in‚0 disk.rbd2.out‚ 628.9296875 cpu‚25.335855899165335 disk.sda.in‚0.5625Ïzd62Ë%- _wandbruntime‚8388  _runtime‚8388z¥¹TÆÌ%À _step‚808 & lr-AdamW/pg1‚3.916236145222388e-05 & lr-AdamW/pg2‚3.916236145222388e-05 trainer/global_step‚9849 $ _timestamp‚1.788445948185149e+09 _runtime‚8388.229875371dª¼/ËÍ%Å & lr-AdamW/pg1‚3.916236145222388e-05 & lr-AdamW/pg2‚3.916236145222388e-05 trainer/global_step‚9849 $ _timestamp‚1.788445948185149e+09 _runtime‚8388.229875371 _step‚808¨sŒø< Î%- _wx9Éã'andbruntime‚8388  _runtime‚83881²{(Ï%–P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚966352335631872 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚966352261101056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚941316476831744 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚8388.230913697 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94579652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94579649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚941316504966144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1118707457265664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18261138773968 trainer/global_step‚9849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145722727 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18288879941632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18258540489300 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25035784269312 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183911055 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145722733 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1101035944514512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1118704830664704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68386676 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145722733 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183931089 % _timestamp‚1.7884459481859984e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 _step‚809 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68386231 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1082774805740544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1118707457265664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1100415950723072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1100415950723072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183931089 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18291480151040 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1082774780312576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1118704830664704 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162965880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18288879941632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18291480151040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25035830665728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448µ~Û!(Ð%›P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18258540489300 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25035784269312 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183911055 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145722733 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1101035944514512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1118704830664704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68386676 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145722733 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183931089 % _timestamp‚1.7884459481859984e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 _step‚809 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68386231 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1082774805740544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1118707457265664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1100415950723072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1100415950723072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183931089 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18291480151040 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1082774780312576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1118704830664704 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162965880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18288879941632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18291480151040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25035830665728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚966352335631872 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚966352261101056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚941316476831744 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _runtime‚8388.230913697 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94579652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94579649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚941316504966144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1118707457265664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18261138773968 trainer/global_step‚9849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145722727 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18288879941632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822©˜Ø2Ñ%- _wandbruntime‚8389  _runtime‚8389Óu &'Ò% N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1100523890428416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18290712270336 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183929289 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚417 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚966447182602240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚941408864885248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18262968116844 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1100523916819968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329666252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1100523890428416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚8389.004764194 _step‚810 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1101141491318816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25038317716992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94588848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145736963 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183949349 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1100523916819968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25038362002944 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1082881121486848 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18293312493056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚966447255022592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚941408893019648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94588845 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145736969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1118817229313024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1101144115031660 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162981729 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68392884 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183949349 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1100523916819968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25038362002944 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1082881121486848 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18293312493056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚966447255022592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚941408893019648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 PADeviý¡2‹ceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94588845 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145736969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1118817229313024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1101144115031660 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162981729 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68392884 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚966447182602240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚941408864885248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18262968116844 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1100523916819968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329666252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1100523890428416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚8389.004764194 _step‚810 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1101141491318816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25038317716992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94588848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145736963 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _step‚813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1124196073381888 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1088091864454144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1105812962381824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1124196073381888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25162402125312 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1105812935990272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18380510790656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68722694 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146434527 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184846141 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163762143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1124193446780928 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1105812962381824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1124193446780928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163762607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184826107 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146434527 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚945935907641344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1105812935990272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18380510790656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68722694 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146434527 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184846141 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163762143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1124193446780928 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1105812962381824 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1124193446780928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163762607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184826107 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146434527 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 eQDeviceStatsMonito‚‘É‘é r.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚945935907641344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _step‚813 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1124196073381888 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1088091864454144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1105812962381824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1124196073381888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0­q32ã%- _wandbruntime‚8430  _runtime‚8430òßî&'ä% N PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163778386 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1105920875695616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95048648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68729299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚946028267560448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1088198205628416 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1124305845429248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95048645 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1124305845429248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184844341 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚946028295694848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18351850556576 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _runtime‚8430.845081568 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105920902087168 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1105920902087168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚971193252371968 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18382343119360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146448769 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25164908196864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146448763 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1124303218814976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184864401 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25164956677120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18384943342080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163777944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18382343119360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146448763 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 trainer/global_step‚9899 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18354448841452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚971193175757312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 % _timestamp‚1.7884459907997525e+09 _step‚814 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184864401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18382343119360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146448769 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25164908196864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146448763 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1124303218814976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184864401 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25164956677120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18384943342080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163777944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18382343119360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146448763 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 trainer/global_step‚9899 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18354448841452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚971193175757312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 % _timestamp‚1.7884459907997525e+09 _step‚814 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184864401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332887486 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185761193 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18474741849088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185761193 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1093408897739776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1129682062897152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185741159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1111850399678292 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164558741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95499252 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907526 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25288702247424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147146327 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649»8j!(ò%›P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147146327 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚975843984429568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1129684689498112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164558286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1093408923167744 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332887486 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185761193 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18474741849088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185761193 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1093408897739776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1129682062897152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185741159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1111850399678292 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164558741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95499252 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907526 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25288702247424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0±‡éù2ó%- _wandbruntime‚8472  _runtime‚8472Ãd°;&'ô% N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185759393 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1129794461545472 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164574076 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185779453 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185779453 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95508448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69066056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18443331281184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919956 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1111317887354368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1111961193908076 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1093515238914048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1111317860962816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18473973968384 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1129791834931200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚950647670235648 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147160569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940022 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147160563 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95508445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69065631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25291275620352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚975938899472896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18476574191104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1111317887354368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚950647698370048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 _runtime‚8472.334553999 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25291229237248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1093515264342016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18445929566060 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÆwÚõ%¥N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18443331281184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919956 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1111317887354368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1111961193908076 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚425 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1093515238914048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1111317860962816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18473973968384 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1129791834931200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚950647670235648 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147160569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940022 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147160563 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95508445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69065631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25291275620352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XDDeviceStatsMonitor.oc†’„Qn_train_batch_end/inactive_split_bytes.all.freed‚975938899472896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18476574191104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1111317887354368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚950647698370048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 _runtime‚8472.334553999 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25291229237248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1093515264342016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18445929566060 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185759393 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1129794461545472 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164574076 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185779453 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185779453 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95508448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69066056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400²B&2ö%- _wandbruntime‚8472  _runtime‚8472Ð1þ1[÷%Õ 2loss/SRqqt1/assignment_loss‚0.7459237027159404 0loss/FBt1/assignment_loss‚0.1315659107886368 1loss/SRqqt2/detection_loss‚1.7064468565773456 2loss/SRqqt4/assignment_loss‚1.2139907352259902 .loss/FBt4/detection_loss‚2.280141044121522 _runtime‚8472.335709086 _step‚819 /loss/FRt1/assignment_loss‚1.165155053138733 0loss/FBt2/assignment_loss‚0.2836304181982996 .loss/FRt1/detection_loss‚1.411905248608926 /loss/FRt3/detection_loss‚2.3483121020757167 $ _timestamp‚1.788446032291114e+09 2loss/SRqqt2/assignment_loss‚1.3220513776103233 1loss/FBt3/assignment_loss‚0.07159340256233311 /loss/FRt4/detection_loss‚2.0135808597176763 1loss/SRqqt1/detection_loss‚0.7541566292149207 .loss/FBt2/detection_loss‚1.992663065592448 0loss/FRt4/assignment_loss‚2.2476320590003063 2loss/SRqqt3/assignment_loss‚1.5418507923421108 0loss/FRt2/assignment_loss‚2.1976380894581475 0loss/FRt3/assignment_loss‚2.7220143820788407 /loss/FRt2/detection_loss‚2.3870429769158363 0loss/SRqqt3/detection_loss‚2.227775146302424 0loss/FBt1/detection_loss‚0.46910155886351795 epoch‚8 trainer/global_step‚9949 0loss/FBt4/assignment_loss‚0.3651883916665191 1loss/SRqqt4/detection_loss‚1.4967258193942368 /loss/FBt3/detection_loss‚3.2092451254526773 %loss/total_loss‚36.30533074762343ì`ø%Ú 1loss/SRqqt1/detection_loss‚0.7541566292149207 .loss/FBt2/detection_loss‚1.992663065592448 0loss/FRt4/assignment_loss‚2.2476320590003063 2loss/SRqqt3/assignment_loss‚1.5418507923421108 0loss/FRt2/assignment_loss‚2.1976380894581475 0loss/FRt3/assignment_loss‚2.7220143820788407 /loss/FRt2/detection_loss‚2.3870429769158363 0loss/SRqqt3/detection_loss‚2.227775146302424 0loss/FBt1/detection_loss‚0.46910155886351795 epoch‚8 trainer/global_step‚9949 0loss/FBt4/assignment_loss‚0.3651883916665191 1loss/SRqqt4/detection_loss‚1.4967258193942368 /loss/FBt3/detection_loss‚3.2092451254526773 %loss/total_loss‚36.30533074762343 2loss/SRqqt1/assignment_loss‚0.7459237027159404 0loss/FBt1/assignment_loss‚0.1315659107886368 1loss/SRqqt2/detection_loss‚1.7064468565773456 2loss/SRqqt4/assignment_loss‚1.2139907352259902 .loss/FBt4/detection_loss‚2.280141044121522 _runtime‚8472.335709086 _step‚819 /loss/FRt1/assignment_loss‚1.165155053138733 0loss/FBt2/assignment_loss‚0.2836304181982996 .loss/FRt1/detection_loss‚1.411905248608926 /loss/FRt3/detection_loss‚2.3483121020757167 $ _timestamp‚1.788446032291114e+09 2loss/SRqqt2/assignment_loss‚1.3220513776103233 1loss/FBt3/assignment_loss‚0.07159340256233311 /loss/FRt4/detection_loss‚2.0135808597176763³ö•†,“ù%‚(:ˆ ÒŠæÔÌÖ—É gpu.0.gpu‚48.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚265.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.079 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³¸Û4ú%‚(:© ÒŠæÔ–§šÌ% disk./.usageGB‚1108.3375205993652 disk.sda1.in‚0 disk.rbd1.in‚0! disk.rbd2.out‚630.81689453125 cpu‚25.3328518615641 network.sent‚ 2.5123e+07) disk./.usagePercent‚74.38146096454463 disk.dm-1.out‚ 307.26171875& disk.nvme0n1.out‚73777.46240234375 disk.dm-0.in‚ 1760.1875 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 1760.1875# disk.dm-0.out‚73777.46240234375 disk.sda.in‚0.5625 disk.sda3.out‚ 307.26171875 network.recv‚ 3.092646e+06 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1760.1875 disk.sda1.out‚0( disk.nvme0c0n1.out‚73777.46240234375! proc.memory.rssMB‚ 2725.609375 proc.cpu.threads‚17 disk.sda.out‚ 307.26171875 disk.rbd1.out‚ 4.35546875 disk.sda3.in‚0.5625$ memory_percent‚77.13935375213623) proc.memory.availableMB‚ 3745.48828125* proc.memory.percent‚16.635799407958984€»¼¶”û%‚(:‰ áŠæÔ£åÎÈ gpu.0.gpu‚44.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚218.417' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.6834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ró2ü%‚(:§ áŠæÔ짎Ì disk.sda2.out‚0 network.recv‚ 3.099374e+06 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1760.1875 disk.dm-1.out‚ 308.01171875) disk./.usagePercent‚74.38143049964806! disk.rbd2.out‚632.46923828125 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚73778.416015625 cpu‚25.33238364233096 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875$ disk.nvme0n1.out‚73778.416015625 proc.cpu.threads‚17 network.sent‚ 2.5232056e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 308.01171875! disk.dm-0.out‚73778.416015625* proc.memory.percent‚16.635799407958984 disk.nvme0c0n1.in‚ 1760.1875 disk.dm-1.in‚0.5625) proc.memory.availableMB‚ 1872.05859375! proc.memory.rssMB‚ 2725.609375 disk.drbd1777.out‚0 disk.sda.out‚ 308.01171875 disk.nvme0n1.in‚ 1760.1875$ memory_percent‚88.57386112213135% disk./.usageGB‚1108.3370666503906 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda3.in‚0.5625hØ,[”ý%‚(:‰ ðŠæÔÔ¦öÈ gpu.0.gpu‚91.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚264.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ%t-þ%‚(:¢ ðŠæÔªÖ÷Ë$ memory_percent‚88.57624530792236 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1760.1875 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 1760.1875) proc.memory.availableMB‚ 1871.66796875* proc.memory.percent‚16.635799407958984 cpu‚25.331888264339238 disk.rbd1.out‚ 4.35546875 network.sent‚ 2.5237803e+07 network.recv‚ 3.103889e+06! disk.rbd2.out‚632.46923828125 disk.sda.out‚ 308.328125 disk.nvme0n1.in‚ 1760.1875 disk.dm-1.out‚ 308.328125 disk.rbd1.in‚0 disk.dm-1.in‚0.5625! proc.memory.rssMB‚ 2725.609375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 308.328125! disk.dm-0.out‚73875.908203125) disk./.usagePercent‚74.38140720296245 disk.drbd1777.out‚0 disk.sda.in‚0.5625$ disk.nvme0n1.out‚73875.908203125& disk.nvme0c0n1.out‚73875.908203125% disk./.usageGB‚1108.3367195129395 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚05ðp¾2ÿ%- _wandbruntime‚8512  _runtime‚8512¾doSÇ€&Á & lr-AdamW/pg2‚3.425047029537249e-05 trainer/global_step‚9999 % _timestamp‚1.7884460724201066e+09 _runtime‚8512.464795364 _step‚820 & lr-AdamW/pg1‚3.425047029537249e-05e1ÀÌ&Æ trainer/global_step‚9999 % _timestamp‚1.7884460724201066e+09 _runtime‚8512.464795364 _step‚820 & lr-AdamW/pg1‚3.425047029537249e-05 & lr-AdamW/pg2‚3.425047029537249e-05´"(3û2‚&- _wandbruntime‚8512  _runtime‚8512‰úý(ƒ&–P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186676245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1135173305614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _runtime‚8512.466282838 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186656211 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚955174684857344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1135170679013376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1116606932916224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25414979863552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1116606906524672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69394954 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1098725981881344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1135173305614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186656211 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18566372698112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165354003 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1098725956453376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147858133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95959052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95959049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚980589618324480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25414933467136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165354488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1135170679013376a¢ñ$„&›P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69394954 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1098725981881344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1135173305614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186656211 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18566372698112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165354003 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1098725956453376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147858133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95959052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95959049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚980589618324480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25414933467136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165354488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1135170679013376 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186676245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1135173305614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _runtime‚8512.466282838 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186656211 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚955174684857344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1135170679013376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1116606932916224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25414979863552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1116606906524672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575µ â9ó2…&- _wandbruntime‚8513  _runtime‚8513ÿëÉ«&'†& N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147872369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69401538 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18534812005792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1135283077661696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚980684519573504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18537410290668 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95968248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1098832323055616 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1116714846230016 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18565604817408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165369783 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147872363 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95968245 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1117369733346284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147872369 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚955267101045248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334546808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147872363 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1098832297627648 trainer/global_step‚9999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18568205040128 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186694505 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚8513.203786976 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1116714846230016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚980684594091008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25417493045760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18565604817408 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1117367109633440 _step‚822 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69401995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¥9-ˆ+'‡&¥N trainer/global_step‚9999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18568205040128 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186694505 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚8513.203786976 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1116714846230016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚980684594091008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25417493045760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18565604817408 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1117367109633440 _step‚822 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69401995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147872369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69401538 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18534812005792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1135283077661696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚980684519573504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18537410290668 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95968248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1098832323055616 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1116714846230016 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18565604817408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165369783 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147872363 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95968245 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1117369733346284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147872369 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚955267101045248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334546808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147872363 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1098832297627648¶‹u2ˆ&- _wandbruntime‚8513  _runtime‚8513}!$@W‰&Ñ /loss/FRt3/assignment_loss‚3.239174356063207 /loss/SRqqt3/detection_loss‚2.08395252839939 2loss/SRqqt1/assignment_loss‚0.7521666385504459 .loss/FBt4/detection_loss‚2.451418009671298 epoch‚8 0loss/FRt2/assignment_loss‚1.9538410431460331 0loss/FRt4/assignment_loss‚2.4591499189051187 /loss/FRt2/detection_loss‚2.5476574238977934 1loss/FBt1/assignment_loss‚0.11933289708994835 /loss/FRt1/detection_loss‚1.6171021396864433 ,loss/FRt3/detection_loss‚2.3526665803923 1loss/SRqqt1/detection_loss‚0.9836222137977828 /loss/FBt1/detection_loss‚0.6117802759399638 .loss/FBt2/detection_loss‚2.447869214746687 0loss/FBt2/assignment_loss‚0.4085311680636765 0loss/FBt4/assignment_loss‚0.2607684729912514 1loss/SRqqt4/detection_loss‚1.5242059081792831 % _timestamp‚1.7884460731602979e+09 _step‚823 2loss/SRqqt3/assignment_loss‚1.5500541963690035 1loss/FBt3/assignment_loss‚0.12498510958782087 /loss/FRt4/detection_loss‚2.1869264702971387 %loss/total_loss‚37.77253721603188 /loss/FRt1/assignment_loss‚1.332076219879851 1loss/SRqqt2/detection_loss‚1.8802015496061204 /loss/FBt3/detection_loss‚2.4901172717412314 _runtime‚8513.204873474 1loss/SRqqt2/assignment_loss‚1.075578648279956 2loss/SRqqt4/assignment_loss‚1.3193589607501364 trainer/global_step‚9999©¸£\Š&Ö /loss/FRt1/assignment_loss‚1.332076219879851 1loss/SRqqt2/detection_loss‚1.8802015496061204 /loss/FBt3/detection_loss‚2.4901172717412314 _runtime‚8513.204873474 1loss/SRqqt2/assignment_loss‚1.075578648279956 2loss/SRqqt4/assignment_loss‚1.3193589607501364 trainer/global_step‚9999 /loss/FRt3/assignment_loss‚3.239174356063207 /loss/SRqqt3/detection_loss‚2.08395252839939 2loss/SRqqt1/assignment_loss‚0.7521666385504459 .loss/FBt4/detection_loss‚2.451418009671298 epoch‚8 0loss/FRt2/assignment_loss‚1.9538410431460331 0loss/FRt4/assignment_loss‚2.4591499189051187 /loss/FRt2/detection_loss‚2.5476574238977934 1loss/FBt1/assignment_loss‚0.11933289708994835 /loss/FRt1/detection_loss‚1.6171021396864433 ,loss/FRt3/detection_loss‚2.3526665803923 1loss/SRqqt1/detection_loss‚0.9836222137977828 /loss/FBt1/detection_loss‚0.6117802759399638 .loss/FBt2/detection_loss‚2.447869214746687 0loss/FBt2/assignment_loss‚0.4085311680636765 0loss/FBt4/assignment_loss‚0.2607684729912514 1loss/SRqqt4/detection_loss‚1.5242059081792831 % _timestamp‚1.7884460731602979e+09 _step‚823 2loss/SRqqt3/assignment_loss‚1.5500541963690035 1loss/FBt3/assignment_loss‚0.12498510958782087 /loss/FRt4/detection_loss‚2.1869264702971387 %loss/total_loss‚37.77253721603188·£ÐÙ´‘‹&‚(:† ÿŠæÔøøöÉ gpu.0.gpu‚65.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚254.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'=ò·-Œ&‚(:¢ ÿŠæÔšïëË disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875 disk.sda.in‚0.5625 disk.sda1.in‚0! proc.memory.rssMB‚ 2725.609375 disk.sda.out‚ 309.12109375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 309.12109375 network.sent‚ 2.5244027e+07 disk.dm-0.in‚ 1760.1875 disk.dm-0.out‚ 73876.7109375! disk.rbd2.out‚632.60986328125 disk.sda1.out‚0) proc.memory.availableMB‚ 1871.41796875 disk.nvme0c0n1.in‚ 1760.1875 disk.sda3.out‚ 309.12109375 disk.drbd1777.out‚0 network.recv‚ 3.107948e+06) disk./.usagePercent‚74.38141334714327$ disk.nvme0c0n1.out‚ 73876.7109375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0.5625$ memory_percent‚88.57777118682861 disk.rbd3.in‚0" disk.nvme0n1.out‚ 73876.7109375* proc.memory.percent‚16.635799407958984 cpu‚25.331394068508747% disk./.usageGB‚1108.3368110656738 disk.nvme0n1.in‚ 1760.18756ÇT'ž&‚(:“ Ž‹æÔÀ´›É gpu.0.gpu‚79.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚255.993' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.19859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>'c)Ž&‚(:ž Ž‹æÔ—ˆ†Ë$ disk.nvme0c0n1.out‚ 73877.0390625 disk.dm-1.out‚ 309.59765625! disk.rbd2.out‚632.60986328125 disk.sda3.in‚0.5625 disk.nvme0c0n1.in‚ 1760.1875 disk.sda2.in‚0 disk.sda3.out‚ 309.59765625 cpu‚25.3309060080335 network.sent‚ 2.5351597e+07 disk.dm-0.out‚ 73877.0390625 disk.dm-1.in‚0.5625 disk.rbd0.out‚0 disk.dm-0.in‚ 1760.1875 disk.rbd1.in‚0 disk.sda2.out‚0# memory_percent‚88.5779857635498! proc.memory.rssMB‚ 2725.609375 proc.cpu.threads‚17% disk./.usageGB‚1108.3368339538574 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 network.recv‚ 3.114008e+06 disk.sda.in‚0.5625 disk.sda.out‚ 309.59765625 disk.rbd0.in‚0* proc.memory.percent‚16.635799407958984) disk./.usagePercent‚74.38141488318847 disk.sda1.in‚0( proc.memory.availableMB‚ 1871.3828125 disk.rbd1.out‚ 4.35546875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 1760.1875" disk.nvme0n1.out‚ 73877.0390625?P N“&‚(:ˆ ‹æÔöϪÉ gpu.0.gpu‚67.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚260.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©^¥†7&‚(:¬ ‹æÔðëÊ disk.rbd3.out‚0# disk.dm-0.out‚73974.71142578125 disk.dm-1.out‚ 310.26953125) proc.memory.availableMB‚ 1871.37890625! proc.memory.rssMB‚ 2725.609375 disk.sda2.out‚0( disk.nvme0c0n1.out‚73974.71142578125 cpu‚25.330445116312656 disk.dm-0.in‚ 1760.1875 disk.rbd2.out‚632.7158203125$ memory_percent‚88.57800960540771 disk.sda.out‚ 310.26953125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.5357603e+07) disk./.usagePercent‚74.38141513919601& disk.nvme0n1.out‚73974.71142578125 disk.nvme0c0n1.in‚ 1760.1875 disk.drbd1777.in‚0% disk./.usageGB‚1108.3368377685547 disk.sda3.in‚0.5625 disk.rbd1.out‚ 4.35546875 proc.cpu.threads‚17 disk.sda.in‚0.5625 disk.sda3.out‚ 310.26953125 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd2.in‚0 network.recv‚ 3.11857e+06 disk.nvme0n1.in‚ 1760.1875 disk.rbd0.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.635799407958984 disk.sda2.in‚0 disk.rbd3.in‚0DIàM2‘&- _wandbruntime‚8553  _runtime‚8553~xÊ’&Ä trainer/global_step‚10049 % _timestamp‚1.7884461135897963e+09 _runtime‚8553.634458112 _step‚824 ' lr-AdamW/pg1‚3.2679206985190825e-05 ' lr-AdamW/pg2‚3.2679206985190825e-05²±ïóÏ“&É % _timestamp‚1.7884461135897963e+09 _runtime‚8553.634458112 _step‚824 ' lr-AdamW/pg1‚3.2679206985190825e-05 ' lr-AdamW/pg2‚3.2679206985190825e-05 trainer/global_step‚10049¸ØÐK’2”&- _wandbruntime‚8553  _runtime‚8553Æpqž•&—P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚959794115666944 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1140661921730560 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1122003918183424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18655403337728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187591297 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1122670102267344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1140659295129600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141190 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25541124700672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336161230 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148569933 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]HDeviceStatsMonitor.on_train_batch_start/allocateb’¿Ðd_bytes.large_pool.freed‚1122003891791872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚959794087532544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚8553.635424113 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148569927 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187571263 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166149322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚985335212233216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187571263 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1122003918183424 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148569933 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187591297 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18624463387732 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1140661921730560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1140659295129600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1122667478554708 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1122003891791872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69730924 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18627061672400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18658003547136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96418849 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148569927 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10049 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25541171097088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18658003547136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1104043015166976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1104043040594944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96418852 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚985335286764032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1122003891791872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚959794087532544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚8553.635424113 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148569927 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187571263 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166149322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚985335212233216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187571263 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1122003918183424 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148569933 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187591297 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18624463387732 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1140661921730560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1140659295129600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1122667478554708 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1122003891791872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69730924 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18627061672400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18658003547136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96418849 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148569927 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10049 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25541171097088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18658003547136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1104043015166976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1104043040594944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96418852 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚985335286764032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚959794115666944 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1140661921730560 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1122003918183424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18655403337728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187591297 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1122670102267344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1140659295129600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336141190 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25541124700672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400¹œ”ò§2—&- _wandbruntime‚8554  _runtime‚8554Âà¯''˜&¡N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚959886503720448 % _timestamp‚1.7884461143488755e+09 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148584163 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18657235666432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69737060 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148584169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚985430129239040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166165105 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148584169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187609557 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187589497 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1140771693777920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚826 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1122111857888768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18657235666432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166165555 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18659835889152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187589497 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104149381769216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18659835889152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1140769067163648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1122111831497216>Snjã™&¦N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166165105 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148584169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187609557 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187589497 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1140771693777920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚826 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1122111857888768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18657235666432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166165555 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18659835889152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187589497 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104149381769216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18659835889152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1140769067163648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1122111831497216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚959886503720448 % _timestamp‚1.7884461143488755e+09 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148584163 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18657235666432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69737060 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148584169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚985430129239040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0º`¾Ù2š&- _wandbruntime‚8554  _runtime‚85541>$}]›&× 2loss/SRqqt3/assignment_loss‚1.1901779232753649 1loss/FBt2/assignment_loss‚0.14016732422169298 /loss/FRt4/detection_loss‚2.6933464392235407 trainer/global_step‚10049 _step‚827 1loss/FBt1/assignment_loss‚0.09350391015039375 1loss/SRqqt1/detection_loss‚0.8793698344911848 1loss/SRqqt2/detection_loss‚1.5274178590024672 %loss/total_loss‚35.18288171320742 /loss/FRt4/assignment_loss‚2.236104973052677 /loss/FRt2/detection_loss‚2.3271038763700647 /loss/FBt2/detection_loss‚2.2990791286740984 /loss/FBt4/detection_loss‚2.0526466012001037 0loss/FRt2/assignment_loss‚1.8795790811802477 /loss/FBt1/detection_loss‚0.6442777958892092 .loss/FBt3/detection_loss‚2.133188954421452 /loss/FRt3/assignment_loss‚2.968481551198398 /loss/FRt3/detection_loss‚2.4325993609773455 0loss/FBt3/assignment_loss‚0.1523682213903937 /loss/FRt1/detection_loss‚1.4694392647513228 2loss/SRqqt2/assignment_loss‚0.9908456808451803 2loss/SRqqt4/assignment_loss‚1.0862810675386099 epoch‚8 $ _timestamp‚1.788446114350776e+09 0loss/FRt1/assignment_loss‚0.6700692384139352 1loss/SRqqt1/assignment_loss‚0.652165866111006 1loss/FBt4/assignment_loss‚0.31447829185053705 0loss/SRqqt3/detection_loss‚2.736133183594103 1loss/SRqqt4/detection_loss‚1.6140562853840894 _runtime‚8554.395376197ˆª4bœ&Ü 2loss/SRqqt3/assignment_loss‚1.1901779232753649 1loss/FBt2/assignment_loss‚0.14016732422169298 /loss/FRt4/detection_loss‚2.6933464392235407 trainer/global_step‚10049 _step‚827 1loss/FBt1/assignment_loss‚0.09350391015039375 1loss/SRqqt1/detection_loss‚0.8793698344911848 1loss/SRqqt2/detection_loss‚1.5274178590024672 %loss/total_loss‚35.18288171320742 /loss/FRt4/assignment_loss‚2.236104973052677 /loss/FRt2/detection_loss‚2.3271038763700647 /loss/FBt2/detection_loss‚2.2990791286740984 /loss/FBt4/detection_loss‚2.0526466012001037 0loss/FRt2/assignment_loss‚1.8795790811802477 /loss/FBt1/detection_loss‚0.6442777958892092 .loss/FBt3/detection_loss‚2.133188954421452 /loss/FRt3/assignment_loss‚2.968481551198398 /loss/FRt3/detection_loss‚2.4325993609773455 0loss/FBt3/assignment_loss‚0.1523682213903937 /loss/FRt1/detection_loss‚1.4694392647513228 2loss/SRqqt2/assignment_loss‚0.9908456808451803 2loss/SRqqt4/assignment_loss‚1.0862810675386099 epoch‚8 $ _timestamp‚1.788446114350776e+09 0loss/FRt1/assignment_loss‚0.6700692384139352 1loss/SRqqt1/assignment_loss‚0.652165866111006 1loss/FBt4/assignment_loss‚0.31447829185053705 0loss/SRqqt3/detection_loss‚2.736133183594103 1loss/SRqqt4/detection_loss‚1.6140562853840894 _runtime‚8554.395376197»Ê®P#”&‚(:‰ ¬‹æÔôŒ¤É gpu.0.gpu‚67.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xˆ4ž&‚(:© ¬‹æÔø¹§Ë# disk.dm-0.out‚73975.13232421875 disk.rbd0.out‚0# memory_percent‚88.5784387588501 proc.cpu.threads‚17( disk.nvme0c0n1.out‚73975.13232421875 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1760.1875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1871.30859375 network.sent‚ 2.5363395e+07) disk./.usagePercent‚74.38151575015696 disk.drbd1777.out‚0 disk.sda.out‚ 310.72265625 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.rbd1.out‚ 4.35546875 disk.rbd2.out‚632.7158203125 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 310.72265625 cpu‚25.32995260537205 network.recv‚ 3.123027e+06 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2725.609375# disk./.usageGB‚1108.33833694458 disk.nvme0c0n1.in‚ 1760.1875 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 310.72265625 disk.rbd1.in‚0* proc.memory.percent‚16.635799407958984 disk.dm-0.in‚ 1760.1875& disk.nvme0n1.out‚73975.13232421875…t’Ÿ&‚(:‡ »‹æÔ¤ûÿÈ gpu.0.gpu‚4.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚253.807' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÂÌ5 &‚(:ª »‹æÔ€šßÊ% disk.nvme0n1.out‚74075.1083984375 proc.cpu.threads‚17 network.recv‚ 3.129643e+06) disk./.usagePercent‚74.38156413558093 disk.rbd1.out‚ 4.35546875 disk.rbd2.out‚634.3056640625$ memory_percent‚88.57839107513428 disk.dm-1.out‚311.5625 disk.drbd1777.out‚0 cpu‚25.329468209000762 network.sent‚ 2.5472792e+07# disk.nvme0c0n1.in‚ 1760.19140625" disk.dm-0.out‚74075.1083984375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚16.635799407958984 disk.rbd0.in‚0' disk.nvme0c0n1.out‚74075.1083984375 disk.drbd1777.in‚0 disk.sda.out‚311.5625! disk.nvme0n1.in‚ 1760.19140625! proc.memory.rssMB‚ 2725.609375% disk./.usageGB‚1108.3390579223633 disk.rbd0.out‚0 disk.dm-0.in‚ 1760.19140625 disk.sda2.out‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0.5625 disk.sda3.out‚311.5625) proc.memory.availableMB‚ 1871.31640625 disk.dm-1.in‚0.5625 disk.rbd3.out‚0Ciï¤2¡&- _wandbruntime‚8593  _runtime‚8593å¬Í4È¢& _step‚828 & lr-AdamW/pg1‚3.114151205544454e-05 & lr-AdamW/pg2‚3.114151205544454e-05 trainer/global_step‚10099 % _timestamp‚1.7884461532544715e+09 _runtime‚8593.299117718Sò(Í£&Ç & lr-AdamW/pg1‚3.114151205544454e-05 & lr-AdamW/pg2‚3.114151205544454e-05 trainer/global_step‚10099 % _timestamp‚1.7884461532544715e+09 _runtime‚8593.299117718 _step‚828¼ßðD{2¤&- _wandbruntime‚8593  _runtime‚8593¦ªM±(¥&–P UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337788082 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1128078641705552 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166946001 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18715944112340 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1146150537846784 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1128076017992916 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚990080736195584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚964413490207744 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1146147911245824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70067349 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1146147911245824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96878649 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18718542397008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1127400877059072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149281733 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚8593.300017164 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188486315 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166945546 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18749634396160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18747034186752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚990080810726400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚964413518342144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚829 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1127400903450624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1127400903450624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1109360099308544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188506349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25667292384256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚10099 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337768042 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1146147911245824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96878649 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18718542397008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1127400877059072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149281733 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 _runtime‚8593.300017164 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188486315 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166945546 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18749634396160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18747034186752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚990080810726400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚964413518342144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚829 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1127400903450624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1127400903450624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1109360099308544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188506349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25667292384256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚10099 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337768042 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18748866515456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _step‚830 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149295963 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149295963 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1146260309894144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1127508816764416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 VIDevice –ÐæÇStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70074013 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188524609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 trainer/global_step‚10099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166961861 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1127508843155968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18751466738176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96887845 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25669770290176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 $ _timestamp‚1.788446154008082e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96887848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1109466440482816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18720371739884 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166961426 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18717773455008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18751466738176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚964505878261248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1109466415054848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820578 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚26491°À+'©&¥N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1127508816764416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70074013 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188524609 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 trainer/global_step‚10099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166961861 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1127508843155968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18751466738176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96887845 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25669770290176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 $ _timestamp‚1.788446154008082e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96887848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1109466440482816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18720371739884 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166961426 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18717773455008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18751466738176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚964505878261248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1109466415054848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820578 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337800512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25669816673280 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1128186812222700 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18748866515456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _step‚830 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149295963 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149295963 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1146260309894144¾@±02ª&- _wandbruntime‚8594  _runtime‚8594“(‰W«&Ñ 0loss/FRt1/assignment_loss‚0.6789952581579035 0loss/FRt2/assignment_loss‚2.6933027112737613 2loss/SRqqt4/assignment_loss‚1.1522001634775239 1loss/FBt1/assignment_loss‚0.07753031497384806 epoch‚8 0loss/FRt3/assignment_loss‚3.0796378600981926 2loss/SRqqt1/assignment_loss‚0.6549820330735071 -loss/FRt2/detection_loss‚2.33919679706401 /loss/FBt1/detection_loss‚0.6602458750425527 .loss/FBt2/detection_loss‚3.090888464450836 $loss/total_loss‚38.3985503477101 2loss/SRqqt2/assignment_loss‚1.4824703813243945 trainer/global_step‚10099 _runtime‚ 8594.05460689 _step‚831 /loss/FRt1/detection_loss‚1.5028773650188338 .loss/FRt4/detection_loss‚2.414368513018586 1loss/SRqqt2/detection_loss‚1.8484280996311169 1loss/SRqqt3/detection_loss‚2.2387683361768724 /loss/FRt4/assignment_loss‚2.566454739071602 1loss/SRqqt1/detection_loss‚0.9385585519211257 1loss/FBt3/assignment_loss‚0.17674363409501634 2loss/SRqqt3/assignment_loss‚1.4247586630284785 0loss/FBt2/assignment_loss‚0.0500900064711459 .loss/FRt3/detection_loss‚2.785632864580966 -loss/FBt3/detection_loss‚2.25660582951137 $ _timestamp‚1.788446154010003e+09 1loss/FBt4/assignment_loss‚0.08679931478885312 1loss/SRqqt4/detection_loss‚1.8190905191681601 .loss/FBt4/detection_loss‚2.379924052291446í žX\¬&Ö $loss/total_loss‚38.3985503477101 2loss/SRqqt2/assignment_loss‚1.4824703813243945 trainer/global_step‚10099 _runtime‚ 8594.05460689 _step‚831 /loss/FRt1/detection_loss‚1.5028773650188338 .loss/FRt4/detection_loss‚2.414368513018586 1loss/SRqqt2/detection_loss‚1.8484280996311169 1loss/SRqqt3/detection_loss‚2.2387683361768724 /loss/FRt4/assignment_loss‚2.566454739071602 1loss/SRqqt1/detection_loss‚0.9385585519211257 1loss/FBt3/assignment_loss‚0.17674363409501634 2loss/SRqqt3/assignment_loss‚1.4247586630284785 0loss/FBt2/assignment_loss‚0.0500900064711459 .loss/FRt3/detection_loss‚2.785632864580966 -loss/FBt3/detection_loss‚2.25660582951137 $ _timestamp‚1.788446154010003e+09 1loss/FBt4/assignment_loss‚0.08679931478885312 1loss/SRqqt4/detection_loss‚1.8190905191681601 .loss/FBt4/detection_loss‚2.379924052291446 0loss/FRt1/assignment_loss‚0.6789952581579035 0loss/FRt2/assignment_loss‚2.6933027112737613 2loss/SRqqt4/assignment_loss‚1.1522001634775239 1loss/FBt1/assignment_loss‚0.07753031497384806 epoch‚8 0loss/FRt3/assignment_loss‚3.0796378600981926 2loss/SRqqt1/assignment_loss‚0.6549820330735071 -loss/FRt2/detection_loss‚2.33919679706401 /loss/FBt1/detection_loss‚0.6602458750425527 .loss/FBt2/detection_loss‚3.090888464450836¿E2^÷ž­&‚(:“ Ê‹æÔ»¶¬É gpu.0.gpu‚81.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚93.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.717200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XD²@®&‚(:µ Ê‹æÔ‰é«Ë disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2725.609375 network.sent‚ 2.5477996e+07 disk.sda3.out‚ 312.16015625* proc.memory.percent‚16.635799407958984 proc.cpu.threads‚17 disk.sda.out‚ 312.16015625 cpu‚25.32897834291263 network.recv‚ 3.133374e+06% disk./.usageGB‚1108.3393363952637# disk.nvme0c0n1.in‚ 1760.19140625" disk.dm-0.out‚74075.8623046875 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 312.16015625 disk.rbd2.out‚634.3056640625 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚74075.8623046875 disk.rbd1.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1760.19140625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚74075.8623046875 disk.drbd1777.out‚0$ memory_percent‚88.57848644256592) proc.memory.availableMB‚ 1871.30078125 disk.rbd1.out‚ 4.35546875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1760.19140625) disk./.usagePercent‚74.38158282413093RÜ„%”¯&‚(:‰ Ù‹æÔ¸ªÉ gpu.0.gpu‚79.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚157.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™;Á@°&‚(:µ Ù‹æÔš—‡Ì% disk./.usageGB‚1108.3370666503906 disk.dm-0.in‚ 1760.19921875 disk.sda2.out‚0% disk.nvme0n1.out‚74176.6181640625 disk.sda3.in‚0.5625* proc.memory.percent‚16.635799407958984 network.sent‚ 2.558721e+07' disk.nvme0c0n1.out‚74176.6181640625" disk.dm-0.out‚74176.6181640625 disk.rbd2.in‚0 disk.sda.out‚ 313.05078125 disk.sda3.out‚ 313.05078125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 4.35546875 disk.sda.in‚0.5625 disk.sda1.in‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1760.19921875 disk.dm-1.out‚ 313.05078125 disk.rbd2.out‚634.3837890625) proc.memory.availableMB‚ 1871.27734375! proc.memory.rssMB‚ 2725.609375 disk.rbd0.in‚0 network.recv‚ 3.140398e+06 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0$ memory_percent‚88.57862949371338 disk.rbd3.out‚0 disk.dm-1.in‚0.5625 cpu‚25.328489520218206) disk./.usagePercent‚74.38143049964806! disk.nvme0n1.in‚ 1760.19921875 disk.drbd1777.in‚0#’±&‚(:‡ è‹æÔ­¡¨É gpu.0.gpu‚68.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚235.44' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼öcu4²&‚(:© è‹æÔíÇÚË disk.dm-1.out‚313.8125 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚634.3837890625! disk.nvme0n1.in‚ 1762.78515625) disk./.usagePercent‚74.38134140902615# disk.nvme0c0n1.in‚ 1762.78515625 disk.sda.in‚0.5625 disk.sda3.out‚313.8125% disk./.usageGB‚1108.3357391357422 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚74177.6806640625 disk.rbd0.out‚0( proc.memory.availableMB‚ 1871.5234375 network.sent‚ 2.5592985e+07 network.recv‚ 3.144625e+06 disk.rbd3.out‚0$ memory_percent‚88.57712745666504 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.635799407958984' disk.nvme0c0n1.out‚74177.6806640625 disk.dm-0.in‚ 1762.78515625" disk.dm-0.out‚74177.6806640625 disk.rbd1.out‚ 4.35546875 disk.sda.out‚313.8125 disk.sda3.in‚0.5625 proc.cpu.threads‚17! proc.memory.rssMB‚ 2725.609375 cpu‚25.328037650451698Ávù2³&- _wandbruntime‚8633  _runtime‚8633T]¶ÔÈ´& trainer/global_step‚10149 % _timestamp‚1.7884461931629415e+09 _runtime‚8633.207584569 _step‚832 & lr-AdamW/pg1‚2.963770595945231e-05 & lr-AdamW/pg2‚2.963770595945231e-05 c͵&Ç & lr-AdamW/pg1‚2.963770595945231e-05 & lr-AdamW/pg2‚2.963770595945231e-05 trainer/global_step‚10149 % _timestamp‚1.7884461931629415e+09 _runtime‚8633.207584569 _step‚832À]‡Ý&2¶&- _wandbruntime‚8633  _runtime‚8633; D­H·&–P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 trainer/global_step‚10149 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1151636527362048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚994826472314880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1132797862326272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339414934 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189401367 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132797888717824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97338449 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _runtime‚8633.208791785 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18838665035776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18810023121616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189401367 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149993533 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 $ _timestamp‚1.788446193163842e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18841265245184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18838665035776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70402251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚994826397784064 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1114677132594176 _step‚833 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25793504901120 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1133487181143760 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149993527 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1151636527362048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189421401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132797862326272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18807424836948 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 WBDeviceStatsMonitor.on_train_batch_start/active_WÌm¢Ô bytes.all.allocated‚1151639153963008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚969032892882944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149993533 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1132797888717824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1114677158022144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149993527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70401805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167740254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339394894 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚969032921017344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18841265245184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189421401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167740703 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1133484557431124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968`¸¶!(¸&›P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167740703 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1133484557431124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 trainer/global_step‚10149 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1151636527362048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚994826472314880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1132797862326272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339414934 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189401367 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132797888717824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97338449 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _runtime‚8633.208791785 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18838665035776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18810023121616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189401367 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149993533 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 $ _timestamp‚1.788446193163842e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18841265245184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18838665035776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70402251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚994826397784064 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1114677132594176 _step‚833 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25793504901120 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1133487181143760 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149993527 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1151636527362048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189421401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132797862326272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18807424836948 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1151639153963008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚969032892882944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149993533 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1132797888717824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1114677158022144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149993527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70401805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167740254 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339394894 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚969032921017344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18841265245184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189421401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649Á¸+P2¹&- _wandbruntime‚8633  _runtime‚8633;<&'º& N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1114783499196416 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1114783473768448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚994921385333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚969125280936448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167756488 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1151746299396096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132905802031616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18843097587200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18840497364480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150007763 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132905828423168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150007763 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189419601 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167756049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70408840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97347648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚994921310815744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1132905802031616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18811852464492 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚834 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70408404 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚969125309070848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 % _timestamp‚1.7884461939185216e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150007769 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189439661 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1151748926010368 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132905828423168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150007769 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25796029879296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18809254179616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376ëà‰¥"»&¥N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189439661 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1151748926010368 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132905828423168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150007769 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25796029879296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18809254179616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1114783499196416 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1114783473768448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚994921385333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚969125280936448 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167756488 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1151746299396096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132905802031616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18843097587200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18840497364480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150007763 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132905828423168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150007763 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189419601 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167756049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70408840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97347648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚994921310815744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1132905802031616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18811852464492 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _step‚834 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70408404 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚969125309070848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 % _timestamp‚1.7884461939185216e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150007769 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376Â$²ûB2¼&- _wandbruntime‚8633  _runtime‚8633Zîe´_½&Ù epoch‚8 2loss/SRqqt2/assignment_loss‚1.0686911542150306 2loss/SRqqt3/assignment_loss‚1.1299611920326509 0loss/FBt2/assignment_loss‚0.0578525935508272 /loss/FRt4/detection_loss‚2.3666215329556852 1loss/SRqqt3/detection_loss‚2.3865139738601795 %loss/total_loss‚37.71880256151346 0loss/FRt2/assignment_loss‚2.5955311088334945 1loss/FBt3/assignment_loss‚0.20295716763939708 /loss/FRt3/detection_loss‚2.2892380132524965 _runtime‚8633.964643623 1loss/FBt1/assignment_loss‚0.03542869956252732 /loss/FBt1/detection_loss‚0.4803866130322308 /loss/FRt2/detection_loss‚2.5269671217316674 trainer/global_step‚10149 $ _timestamp‚1.788446193920065e+09 2loss/SRqqt4/assignment_loss‚1.1771554157752147 .loss/FBt3/detection_loss‚2.782407119870186 0loss/FRt3/assignment_loss‚3.1316653291384378 1loss/FBt4/assignment_loss‚0.03928487587420802 1loss/SRqqt1/detection_loss‚0.7753993819507075 1loss/SRqqt2/detection_loss‚1.8392885205585783 .loss/FBt4/detection_loss‚2.364613558564867 /loss/FRt1/detection_loss‚1.4697347195344228 /loss/FBt2/detection_loss‚2.1829891840616864 _step‚835 0loss/FRt1/assignment_loss‚1.0746746939771317 /loss/FRt4/assignment_loss‚2.850879166577313 2loss/SRqqt1/assignment_loss‚1.0352168740822711 1loss/SRqqt4/detection_loss‚1.8553445508822481gQ%éd¾&Þ 1loss/SRqqt1/detection_loss‚0.7753993819507075 1loss/SRqqt2/detection_loss‚1.8392885205585783 .loss/FBt4/detection_loss‚2.364613558564867 /loss/FRt1/detection_loss‚1.4697347195344228 /loss/FBt2/detection_loss‚2.1829891840616864 _step‚835 0loss/FRt1/assignment_loss‚1.0746746939771317 /loss/FRt4/assignment_loss‚2.850879166577313 2loss/SRqqt1/assignment_loss‚1.0352168740822711 1loss/SRqqt4/detection_loss‚1.8553445508822481 epoch‚8 2loss/SRqqt2/assignment_loss‚1.0686911542150306 2loss/SRqqt3/assignment_loss‚1.1299611920326509 0loss/FBt2/assignment_loss‚0.0578525935508272 /loss/FRt4/detection_loss‚2.3666215329556852 1loss/SRqqt3/detection_loss‚2.3865139738601795 %loss/total_loss‚37.71880256151346 0loss/FRt2/assignment_loss‚2.5955311088334945 1loss/FBt3/assignment_loss‚0.20295716763939708 /loss/FRt3/detection_loss‚2.2892380132524965 _runtime‚8633.964643623 1loss/FBt1/assignment_loss‚0.03542869956252732 /loss/FBt1/detection_loss‚0.4803866130322308 /loss/FRt2/detection_loss‚2.5269671217316674 trainer/global_step‚10149 $ _timestamp‚1.788446193920065e+09 2loss/SRqqt4/assignment_loss‚1.1771554157752147 .loss/FBt3/detection_loss‚2.782407119870186 0loss/FRt3/assignment_loss‚3.1316653291384378 1loss/FBt4/assignment_loss‚0.03928487587420802À£ÒYž¿&‚(:“ ÷‹æÔ€½œÊ gpu.0.gpu‚68.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.993' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.19859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë,Ϭ8À&‚(:­ ÷‹æÔà–ÂË% disk./.usageGB‚1108.3357582092285 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1762.9140625! proc.memory.rssMB‚ 2725.609375* proc.memory.percent‚16.635799407958984$ disk.nvme0n1.out‚74179.107421875& disk.nvme0c0n1.out‚74179.107421875 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 1762.9140625 disk.sda3.in‚0.5625 network.sent‚ 2.5599222e+07 disk.sda.in‚0.5625" disk.nvme0c0n1.in‚ 1762.9140625 cpu‚25.327499935236922 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0$ memory_percent‚88.58017921447754 disk.drbd1777.in‚0( proc.memory.availableMB‚ 1871.0234375) disk./.usagePercent‚74.38134268906381 disk.rbd1.in‚0 disk.sda3.out‚ 314.3984375 disk.dm-1.out‚ 314.3984375 disk.drbd1777.out‚0 network.recv‚ 3.149183e+06! disk.rbd2.out‚634.55615234375 disk.sda.out‚ 314.3984375 disk.sda1.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚74179.107421875í¨Ÿû”Á&‚(:‰ †ŒæÔ ÇÊ gpu.0.gpu‚48.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚268.958' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôö¶£<Â&‚(:± †ŒæÔäÁŒË( disk.nvme0c0n1.out‚74277.29443359375! proc.memory.rssMB‚ 2725.609375 cpu‚25.327051835682052 disk.rbd3.out‚0* proc.memory.percent‚16.635799407958984 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1763.69140625 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0! disk.rbd2.out‚634.55615234375 proc.cpu.threads‚17& disk.nvme0n1.out‚74277.29443359375 disk.sda.out‚ 315.109375 disk.rbd1.out‚ 4.35546875$ memory_percent‚88.57603073120117' proc.memory.availableMB‚ 1871.703125 network.sent‚ 2.5707462e+07 disk.sda.in‚0.5625 disk.sda3.out‚ 315.109375 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 network.recv‚ 3.155159e+06) disk./.usagePercent‚74.38154263094806$ disk./.usageGB‚1108.338737487793 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚74277.29443359375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1763.69140625 disk.dm-1.out‚ 315.109375# disk.nvme0c0n1.in‚ 1763.69140625ÿ{’“Ã&‚(:ˆ •ŒæÔÅÊ›É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚95.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œše68Ä&‚(:­ •ŒæÔ®éÊ disk.sda.in‚0.5625 disk.sda2.in‚0& disk.nvme0n1.out‚74278.89501953125! proc.memory.rssMB‚ 2725.609375 disk.nvme0c0n1.in‚ 1763.9375 disk.rbd1.out‚ 4.35546875 disk.rbd2.in‚0$ memory_percent‚88.57753276824951) proc.memory.availableMB‚ 1871.45703125* proc.memory.percent‚16.635799407958984 network.recv‚ 3.159269e+06( disk.nvme0c0n1.out‚74278.89501953125 disk.sda2.out‚0) disk./.usagePercent‚74.38150806993093# disk.dm-0.out‚74278.89501953125 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1763.9375 disk.dm-1.out‚ 316.04296875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.5713406e+07 disk.drbd1777.in‚0 cpu‚25.326546901892442! disk.rbd2.out‚636.20849609375 disk.sda3.out‚ 316.04296875 disk.sda.out‚ 316.04296875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1763.9375 disk.sda3.in‚0.5625$ disk./.usageGB‚1108.338222503662 disk.dm-1.in‚0.5625g( g2Å&- _wandbruntime‚8673  _runtime‚8673ÃÒúÈÆ& ' lr-AdamW/pg2‚2.8168102088150442e-05 trainer/global_step‚10199 # _timestamp‚1.78844623302046e+09 _runtime‚8673.065127801 _step‚836 ' lr-AdamW/pg1‚2.8168102088150442e-05®u—’ÍÇ&Ç _runtime‚8673.065127801 _step‚836 ' lr-AdamW/pg1‚2.8168102088150442e-05 ' lr-AdamW/pg2‚2.8168102088150442e-05 trainer/global_step‚10199 # _timestamp‚1.78844623302046e+09ÄåÜ@2È&- _wandbruntime‚8673  _runtime‚8673/¹m(É&—P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18898905561556 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1138194873985024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150705327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884462330212033e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70738710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚973652323692544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97798252 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190316419 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚973652295558144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _runtime‚8673.066053165 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚999571917031936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150705327 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1157127770079232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18930295884800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150705333 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190316419 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168536523 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1138194873985024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25919593339392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25919549040128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1157125143478272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18932896094208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18901503846224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚999571844598272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1138895720581968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1138893096869332 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 trainer/global_step‚10199 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1138194847593472 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1138194847593472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168536962 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0È‹Ä"(Ê&œP LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97798252 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190316419 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚973652295558144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 _runtime‚8673.066053165 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚999571917031936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150705327 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1157127770079232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18930295884800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150705333 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190316419 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚44299264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168536523 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1138194873985024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚25919593339392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚25919549040128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1157125143478272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18932896094208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18901503846224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚999571844598272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1138895720581968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1138893096869332 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 trainer/global_step‚10199 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1138194847593472 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1138194847593472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168536962 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18898905561556 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1138194873985024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150705327 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884462330212033e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70738710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚973652323692544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6Å‘«ÙS2Ë&- _wandbruntime‚8673  _runtime‚8673u=  Ì&ŸN KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1138302813690368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚999666827759104 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341054216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168552575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1139001267386272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18934728436224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚973744711746048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1138302813690368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1157234915512320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97807445 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1157237542126592 % _timestamp‚1.7884462337782001e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18903333189100 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150719563 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1138302787298816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150719569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsM¬yŸÎonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚10199 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18900734904224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚999666753241600 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1138302787298816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1120100557910016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _runtime‚ 8673.8233985 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190334653 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1139003891099116 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190354713 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150719563 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25922116013056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168553031 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚973744683611648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25922069629952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341054216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚838 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150719569 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190334653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190354713 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074þ{<*'Í&¤N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1157234915512320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97807445 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1157237542126592 % _timestamp‚1.7884462337782001e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18903333189100 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150719563 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1138302787298816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150719569 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚10199 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18900734904224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚999666753241600 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1138302787298816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1120100557910016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _runtime‚ 8673.8233985 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190334653 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1139003891099116 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190354713 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150719563 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚25922116013056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168553031 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚973744683611648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚25922069629952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341054216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚838 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150719569 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190334653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190354713 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1138302813690368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚999666827759104 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341054216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168552575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1139001267386272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18934728436224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚973744711746048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1138302813690368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560Æ 2rŒ2Î&- _wandbruntime‚8673  _runtime‚8673âO –`Ï&Ú 2loss/FBt2/assignment_loss‚0.006959890095458832 0loss/FBt4/assignment_loss‚0.2790967115821938 /loss/FRt4/detection_loss‚2.3813668827215833 1loss/SRqqt2/detection_loss‚1.6554301712872848 .loss/FBt2/detection_loss‚3.502894848585129 trainer/global_step‚10199 /loss/FRt2/assignment_loss‚2.360252994757432 2loss/SRqqt3/assignment_loss‚1.3790136927908117 /loss/FBt1/detection_loss‚0.5629746211999203 2loss/SRqqt1/assignment_loss‚0.9899522063929136 1loss/FBt3/assignment_loss‚0.10584102956239473 /loss/FRt1/detection_loss‚1.5382779205504518 0loss/FRt1/assignment_loss‚1.3086835088638158 1loss/SRqqt3/detection_loss‚2.1140161894939165 .loss/FBt4/detection_loss‚2.743866960207621 1loss/FBt1/assignment_loss‚0.14579658938065013 /loss/FRt3/detection_loss‚2.9530203761877836 1loss/SRqqt1/detection_loss‚0.9704180957862374 $ _timestamp‚1.788446233780203e+09 _step‚839 2loss/SRqqt4/assignment_loss‚1.4093730199263141 /loss/FBt3/detection_loss‚1.8469872312112288 0loss/FRt4/assignment_loss‚2.3284943964746265 2loss/SRqqt2/assignment_loss‚1.3304842563504178 &loss/total_loss‚38.677073665744516 _runtime‚8673.824781427 0loss/FRt3/assignment_loss‚2.5696009353355125 .loss/FRt2/detection_loss‚2.561924335284111 1loss/SRqqt4/detection_loss‚1.6323468017167058 epoch‚8;~/eÐ&ß /loss/FRt2/assignment_loss‚2.360252994757432 2loss/SRqqt3/assignment_loss‚1.3790136927908117 /loss/FBt1/detection_loss‚0.5629746211999203 2loss/SRqqt1/assignment_loss‚0.9899522063929136 1loss/FBt3/assignment_loss‚0.10584102956239473 /loss/FRt1/detection_loss‚1.5382779205504518 0loss/FRt1/assignment_loss‚1.3086835088638158 1loss/SRqqt3/detection_loss‚2.1140161894939165 .loss/FBt4/detection_loss‚2.743866960207621 1loss/FBt1/assignment_loss‚0.14579658938065013 /loss/FRt3/detection_loss‚2.9530203761877836 1loss/SRqqt1/detection_loss‚0.9704180957862374 $ _timestamp‚1.788446233780203e+09 _step‚839 2loss/SRqqt4/assignment_loss‚1.4093730199263141 /loss/FBt3/detection_loss‚1.8469872312112288 0loss/FRt4/assignment_loss‚2.3284943964746265 2loss/SRqqt2/assignment_loss‚1.3304842563504178 &loss/total_loss‚38.677073665744516 _runtime‚8673.824781427 0loss/FRt3/assignment_loss‚2.5696009353355125 .loss/FRt2/detection_loss‚2.561924335284111 1loss/SRqqt4/detection_loss‚1.6323468017167058 epoch‚8 2loss/FBt2/assignment_loss‚0.006959890095458832 0loss/FBt4/assignment_loss‚0.2790967115821938 /loss/FRt4/detection_loss‚2.3813668827215833 1loss/SRqqt2/detection_loss‚1.6554301712872848 .loss/FBt2/detection_loss‚3.502894848585129 trainer/global_step‚10199ÇáY¨Q’Ñ&‚(:‡ ¤ŒæÔÌͯÉ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚93.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.659 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tCÉR5Ò&‚(:ª ¤ŒæÔ°×¼Ë disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚88.5777473449707' proc.memory.availableMB‚ 1871.421875 cpu‚25.326095468565697 disk.dm-0.in‚ 1764.65625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.sda.in‚0.5625 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 1764.65625* proc.memory.percent‚16.635799407958984) disk./.usagePercent‚74.38152778251107% disk./.usageGB‚1108.3385162353516 disk.rbd1.out‚ 4.35546875 disk.sda2.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1764.65625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚74376.3857421875 disk.dm-1.out‚ 316.74609375 disk.sda3.out‚ 316.74609375 network.recv‚ 3.16365e+06" disk.dm-0.out‚74376.3857421875 disk.drbd1777.in‚0! disk.rbd2.out‚636.20849609375 network.sent‚ 2.5719233e+07 disk.sda.out‚ 316.74609375 disk.sda2.in‚0% disk.nvme0n1.out‚74376.3857421875! proc.memory.rssMB‚ 2725.609375µÆÅÕ”Ó&‚(:‰ ³ŒæÔ¢ãéÈ gpu.0.gpu‚67.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚243.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷DQÕ:Ô&‚(:¯ ³ŒæÔ·¯ÏË disk.sda2.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1871.17578125 cpu‚25.325620071180904 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0$ memory_percent‚88.57924938201904 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 1764.65625% disk./.usageGB‚1108.3383445739746 disk.dm-1.out‚ 317.14453125 disk.nvme0n1.in‚ 1764.65625! proc.memory.rssMB‚ 2725.609375* proc.memory.percent‚16.635799407958984 disk.rbd2.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚74377.15771484375 disk.sda.out‚ 317.14453125 network.sent‚ 2.58286e+07 disk.sda3.in‚0.5625 disk.sda3.out‚ 317.14453125 disk.rbd3.out‚0 disk.dm-0.in‚ 1764.65625# disk.dm-0.out‚74377.15771484375! disk.rbd2.out‚636.31884765625 disk.sda.in‚0.5625 network.recv‚ 3.170353e+06) disk./.usagePercent‚74.38151626217203 disk.rbd1.out‚ 4.35546875 disk.sda1.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚74377.15771484375 disk.sda2.in‚0­X>I2Õ&- _wandbruntime‚8713  _runtime‚8713ØðÊÖ&Ä ' lr-AdamW/pg2‚2.6733006704783094e-05 trainer/global_step‚10249 % _timestamp‚1.7884462730715172e+09 _runtime‚8713.116160659 _step‚840 ' lr-AdamW/pg1‚2.6733006704783094e-05ÌÃè„Ï×&É ' lr-AdamW/pg1‚2.6733006704783094e-05 ' lr-AdamW/pg2‚2.6733006704783094e-05 trainer/global_step‚10249 % _timestamp‚1.7884462730715172e+09 _runtime‚8713.116160659 _step‚840È/¬r72Ø&- _wandbruntime‚8713  _runtime‚8713ÑBâÔÝ$Ù&˜P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98258049 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191251505 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169333115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚978271726367744 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1143591832860672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342668638 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1004317230124032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169332645 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 _step‚841 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98258052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71074596 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18992984570832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1143591859252224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151417127 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26045578287104 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342648598 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151417127 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚2çÛtA4124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71075063 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1162613759594496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968PȆX#(Ú&P [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18990386286164 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚10249 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1162616386195456 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151417133 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1162616386195456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1143591832860672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191251505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26045531890688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1125311275449344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 $ _timestamp‚1.788446273072218e+09 _runtime‚8713.117105008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151417133 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1143591859252224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19024526943232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191231471 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚978271698233344 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1144301636307540 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125311250021376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191231471 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169333115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚978271726367744 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1143591832860672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342668638 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1004317230124032 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169332645 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 _step‚841 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98258052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71074596 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18992984570832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1143591859252224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151417127 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26045578287104 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342648598 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151417127 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71075063 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1162613759594496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98258049 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191251505 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1143699798957568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚8713.882991122 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19023759062528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169348541 _step‚842 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98267248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1162726158242816 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1143699772566016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19026359285248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1162726158242816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191249705 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1125417616623616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 % _timestamp‚1.7884462738377376e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1162723531628544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71081733 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1125417591195648 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1144409806824480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151431363 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1143699772566016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19026359285248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1004412117416448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26048077512704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚05¥\ü.'Ý&¨N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169348541 _step‚842 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98267248 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1162726158242816 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1143699772566016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19026359285248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1162726158242816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191249705 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1125417616623616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 % _timestamp‚1.7884462738377376e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1162723531628544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71081733 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1125417591195648 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1144409806824480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151431363 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1143699772566016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19026359285248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1004412117416448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26048077512704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1143699798957568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚8713.882991122 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19023759062528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ÊÇB@¢2Þ&- _wandbruntime‚8713  _runtime‚8713wv´ïYß&Ó 0loss/SRqqt3/detection_loss‚2.097364222009977 .loss/FBt3/detection_loss‚2.531397372484207 _runtime‚ 8713.88429026 .loss/FBt2/detection_loss‚2.078112210546221 1loss/SRqqt4/assignment_loss‚1.304214988912766 .loss/FRt4/detection_loss‚2.327421290874481 /loss/FRt4/assignment_loss‚2.474916391372681 2loss/SRqqt3/assignment_loss‚1.4694933485136263 1loss/SRqqt2/detection_loss‚1.8754590119363046 1loss/SRqqt4/detection_loss‚1.7152815023437142 trainer/global_step‚10249 1loss/FBt2/assignment_loss‚0.11952414588683626 /loss/FRt1/detection_loss‚1.5586278154700994 .loss/FRt2/detection_loss‚2.468482777890232 &loss/total_loss‚36.397059807077454 /loss/FRt1/assignment_loss‚1.041288081407547 2loss/SRqqt2/assignment_loss‚0.8954798966171402 epoch‚8 /loss/FBt4/detection_loss‚2.7430452009042106 /loss/FRt2/assignment_loss‚2.200060695409775 2loss/SRqqt1/assignment_loss‚0.7937456142330227 0loss/FBt3/assignment_loss‚0.0949945012301517 /loss/FRt3/detection_loss‚2.5991801902957055 1loss/SRqqt1/detection_loss‚0.8869828665081192 /loss/FBt1/detection_loss‚0.4195944184268063 /loss/FRt3/assignment_loss‚2.579966745878521 1loss/FBt4/assignment_loss‚0.05768006199828809 % _timestamp‚1.7884462738396943e+09 1loss/FBt1/assignment_loss‚0.06474645592702907 _step‚843­ö¿Pà&Ø 1loss/FBt4/assignment_loss‚0.05768006199828809 % _timestamp‚1.7884ÍäO462738396943e+09 1loss/FBt1/assignment_loss‚0.06474645592702907 _step‚843 0loss/SRqqt3/detection_loss‚2.097364222009977 .loss/FBt3/detection_loss‚2.531397372484207 _runtime‚ 8713.88429026 .loss/FBt2/detection_loss‚2.078112210546221 1loss/SRqqt4/assignment_loss‚1.304214988912766 .loss/FRt4/detection_loss‚2.327421290874481 /loss/FRt4/assignment_loss‚2.474916391372681 2loss/SRqqt3/assignment_loss‚1.4694933485136263 1loss/SRqqt2/detection_loss‚1.8754590119363046 1loss/SRqqt4/detection_loss‚1.7152815023437142 trainer/global_step‚10249 1loss/FBt2/assignment_loss‚0.11952414588683626 /loss/FRt1/detection_loss‚1.5586278154700994 .loss/FRt2/detection_loss‚2.468482777890232 &loss/total_loss‚36.397059807077454 /loss/FRt1/assignment_loss‚1.041288081407547 2loss/SRqqt2/assignment_loss‚0.8954798966171402 epoch‚8 /loss/FBt4/detection_loss‚2.7430452009042106 /loss/FRt2/assignment_loss‚2.200060695409775 2loss/SRqqt1/assignment_loss‚0.7937456142330227 0loss/FBt3/assignment_loss‚0.0949945012301517 /loss/FRt3/detection_loss‚2.5991801902957055 1loss/SRqqt1/detection_loss‚0.8869828665081192 /loss/FBt1/detection_loss‚0.4195944184268063 /loss/FRt3/assignment_loss‚2.579966745878521Ët¥…Ÿá&‚(:” ÂŒæÔ‡ÝÌÉ gpu.0.gpu‚51.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚264.651' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.930200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øVPj5â&‚(:ª ÂŒæÔÓ¾Ì$ disk./.usageGB‚1108.338249206543 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2725.609375 disk.drbd1777.in‚0 disk.sda.out‚ 317.5234375 disk.rbd0.in‚0$ memory_percent‚88.57913017272949( proc.memory.availableMB‚ 1871.1953125 cpu‚25.32517284531752 disk.rbd1.out‚ 4.35546875 disk.sda3.out‚ 317.5234375 network.recv‚ 3.17376e+06 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1764.65625( disk.nvme0c0n1.out‚74377.64990234375) disk./.usagePercent‚74.38150986198367 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 317.5234375 disk.nvme0c0n1.in‚ 1764.65625 proc.cpu.threads‚17! disk.rbd2.out‚636.31884765625 disk.sda.in‚0.5625 disk.sda2.out‚0& disk.nvme0n1.out‚74377.64990234375 disk.rbd0.out‚0* proc.memory.percent‚16.635799407958984 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 1764.65625# disk.dm-0.out‚74377.64990234375 network.sent‚ 2.5833847e+07ºlëŸã&‚(:” ÑŒæÔÞÊŒÉ gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚143.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.693799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÂIq0ä&‚(:¥ ÑŒæÔ¢»”Ì disk.nvme0c0n1.in‚ 1764.65625 disk.dm-1.out‚ 318.21875! proc.memory.rssMB‚ 2725.609375 network.recv‚ 3.180707e+06 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 318.21875) disk./.usagePercent‚74.38152624646587$ disk./.usageGB‚1108.338493347168 disk.sda.in‚0.5625 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0.5625# disk.dm-0.out‚74475.22412109375( proc.memory.availableMB‚ 1871.1328125 network.sent‚ 2.5942813e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 318.21875( disk.nvme0c0n1.out‚74475.22412109375 cpu‚25.32467260941951 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 1764.65625& disk.nvme0n1.out‚74475.22412109375 disk.rbd0.in‚0 disk.dm-0.in‚ 1764.65625$ memory_percent‚88.57951164245605 disk.rbd2.in‚0 disk.dm-1.in‚0.5625* proc.memory.percent‚16.635799407958984 disk.drbd1777.in‚0! disk.rbd2.out‚636.45556640625 disk.sda1.out‚0 disk.sda2.in‚0 ­¾w”å&‚(:‰ àŒæÔ‡÷È gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚270.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÈ›Q+æ&‚(:  àŒæÔ®“êÊ disk.nvme0c0n1.in‚ 1764.65625 disk.dm-0.in‚ 1764.65625 disk.rbd1.in‚0! disk.rbd2.out‚638.02978515625 disk.sda.in‚0.5625 disk.sda2.out‚0$ disk./.usageGB‚1108.338550567627! disk.dm-0.out‚74476.111328125 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.35546875 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.sda3.in‚0.5625 disk.sda3.out‚ 318.53125 proc.cpu.threads‚17 network.recv‚ 3.184847e+06) disk./.usagePercent‚74.38153008657889 disk.dm-1.out‚ 318.53125 disk.sda.out‚ 318.53125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2725.609375 disk.sda2.in‚0 network.sent‚ 2.5948756e+07$ memory_percent‚88.57810497283936) proc.memory.availableMB‚ 1871.36328125 cpu‚25.32418239049666& disk.nvme0c0n1.out‚74476.111328125 disk.rbd2.in‚0$ disk.nvme0n1.out‚74476.111328125* proc.memory.percent‚16.635799407958984 disk.nvme0n1.in‚ 1764.65625c¨»š2ç&- _wandbruntime‚8752  _runtime‚8752sR¹/Æè&À % lr-AdamW/pg1‚2.53327188810772e-05 % lr-AdamW/pg2‚2.53327188810772e-05 trainer/global_step‚10299 % _timestamp‚1.7884463127772927e+09 _runtime‚8752.821929146 _step‚844qg:ùËé&Å % lr-AdamW/pg1‚2.53327188810772e-05 % lr-AdamW/pg2‚2.53327188810772e-05 trainer/global_step‚10299 % _timestamp‚1.7884463127772927e+09 _runtime‚8752.821929146 _step‚844Ìá\÷ä2ê&- _wandbruntime‚8752  _runtime‚8752ã5›ˆ(ë&˜P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295490 % _timestamp‚1.7884463127782235e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19116157792256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192166557 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1168105002311680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19113557582848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _runtime‚ 8752.82322121 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1148988818127872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1009062206204416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168102375710720 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1148988818127872 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170128873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚845 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192146523 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192166557 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168102375710720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19113557582848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚982891129042944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 trainer/global_step‚10299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1149712799458384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19116157792256 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1148988844519424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170128401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71411021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚469 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19081867010772 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192146523 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1149710175745748 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152128927 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ï#(ì&P TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168102375710720 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1148988818127872 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170128873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚845 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192146523 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192166557 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168102375710720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19113557582848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚982891129042944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 trainer/global_step‚10299 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1149712799458384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19116157792256 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1148988844519424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170128401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71411021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚469 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19081867010772 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192146523 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1149710175745748 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152128927 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295490 % _timestamp‚1.7884463127782235e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19116157792256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192166557 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1168105002311680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19113557582848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _runtime‚ 8752.82322121 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1148988818127872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1009062206204416ÍÊ–ýì2í&- _wandbruntime‚8753  _runtime‚8753¨-^„¿î&£N _step‚846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19117990134272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344307920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170144409 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1168214774359040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19117990134272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192184817 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192164757 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152143163 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192184817 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152143169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1009157117926400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19115389911552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98727045 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168212147744768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚8753.572844121 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152143169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26173628964352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1149820969975532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚‹â‰ªj98727048 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚982983488962048 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71417827 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168214774359040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1149818346262688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19083696353440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327986 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71417364 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 Vƒ.'ï&¨N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170144875 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71417827 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168214774359040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1149818346262688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19083696353440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344327986 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚463 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71417364 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 _step‚846 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19117990134272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344307920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170144409 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1168214774359040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19117990134272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192184817 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192164757 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152143163 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192184817 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152143169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1009157117926400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19115389911552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98727045 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168212147744768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚8753.572844121 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152143169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26173628964352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1149820969975532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98727048 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚982983488962048 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1173590991826944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1154385829786624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1154385829786624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152840727 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193081609 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193061575 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1154385803395072 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19207788641280 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1135945392876544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19173347735380 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152840733 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19207788641280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19175946020048 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1173593618427904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71746664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71746220 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155121338896592 trainer/global_step‚10349 _step‚849 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170923869 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1013807269920256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26296812732928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1135945367448576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788446352303576e+09 _runtime‚ 8792.34843102 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193061575 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170924316 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152840733 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26296766336512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19205188431872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152840727 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚987510531718144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99177652 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚987510503583744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99177649 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1013807344451072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155121338896592 trainer/global_step‚10349 _step‚849 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170923869 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1013807269920256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26296812732928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1135945367448576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788446352303576e+09 _runtime‚ 8792.34843102 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193061575 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170924316 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152840733 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26296766336512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19205188431872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152840727 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚987510531718144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99177652 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚987510503583744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99177649 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1013807344451072 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1173590991826944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1154385829786624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1154385829786624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152840727 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193081609 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193061575 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1154385803395072 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19207788641280 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1135945392876544 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19173347735380 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152840733 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19207788641280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19175946020048 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960ÑP=*2ÿ&- _wandbruntime‚8793  _runtime‚8793ïæÁå)'€'£N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1173703390475264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345934772 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26299332983296 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71753434 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19175177078048 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99186845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1173700763860992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚987602919771648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1013902252754944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1155229509413740 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1136051734050816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193079809 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10349 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19207020760576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚987602891637248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954838 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19209620983296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 _runtime‚8793.087048294 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152854963 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1173703390475264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71752989 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152854969 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193099869 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1136051708622848 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99186848 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1154493743100416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152854969 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193079809 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1154493769491968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1154493743100416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19209620983296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152854963 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170940282 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1154493769491968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _step‚850 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193099869 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1155226885700896 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170939834 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934772EƒÔ.''¨N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193099869 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1155226885700896 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170939834 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934772 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1173703390475264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345934772 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26299332983296 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71753434 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19175177078048 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99186845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1173700763860992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚987602919771648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1013902252754944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1155229509413740 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1136051734050816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193079809 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10349 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19207020760576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚987602891637248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954838 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19209620983296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 _runtime‚8793.087048294 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152854963 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1173703390475264 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71752989 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152854969 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193099869 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1136051708622848 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99186848 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1154493743100416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152854969 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193079809 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1154493769491968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1154493743100416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19209620983296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152854963 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170940282 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1154493769491968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _step‚850Ò¾Q2‚'- _wandbruntime‚8793  _runtime‚8793kÃD{\ƒ'Ö 0loss/SRqqt3/detection_loss‚2.889626408261912 /loss/FRt2/assignment_loss‚2.087867702756609 1loss/FBt2/assignment_loss‚0.09538414665231747 /loss/FRt1/detection_loss‚1.6416907866247172 /loss/FBt3/detection_loss‚2.4559185236692427 % _timestamp‚1.7884463530439522e+09 1loss/FBt3/assignment_loss‚0.08599121242004912 /loss/FBt1/detection_loss‚0.5303817225463126 trainer/global_step‚10349 2loss/SRqqt2/assignment_loss‚1.1477657719789183 1loss/SRqqt4/detection_loss‚1.6208168527689473 _runtime‚8793.088526961 /loss/FRt2/detection_loss‚2.2380495253874333 1loss/SRqqt2/detection_loss‚1.7220756350194706 -loss/FBt4/detection_loss‚2.34882124265035 _step‚851 1loss/FBt4/assignment_loss‚0.37440977192535585 0loss/FRt3/assignment_loss‚3.7036033115888896 1loss/SRqqt3/assignment_loss‚1.926969699561596 /loss/FBt2/detection_loss‚2.8128845606531416 epoch‚8 /loss/FRt1/assignment_loss‚1.408680802167848 0loss/FRt4/assignment_loss‚1.7597186985340985 1loss/SRqqt1/assignment_loss‚0.674805825556078 /loss/FRt3/detection_loss‚2.4545471559502308 /loss/FRt4/detection_loss‚2.1517408906736155 $loss/total_loss‚38.3949582353772 2loss/SRqqt4/assignment_loss‚1.1178328607689012 1loss/FBt1/assignment_loss‚0.11994456935047357 1loss/SRqqt1/detection_loss‚1.0254305579106917¯g.„'Û 2loss/SRqqt2/assignment_loss‚1.1477657719789183 1loss/SRqqt4/detection_loss‚1.6208168527689473 _runtime‚8793.088526961 /loss/FRt2/detection_loss‚2.2380495253874333 1loss/SRqqt2/detection_loss‚1.7220756350194706 -loss/FBt4/detection_loss‚2.34882124265035 _step‚851 1loss/FBt4/assignment_loss‚0.37440977192535585 0loss/FRt3/assignment_loss‚3.7036033115888896 1loss/SRqqt3/assignment_loss‚1.926969699561596 /loss/FBt2/detection_loss‚2.8128845606531416 epoch‚8 /loss/FRt1/assignment_loss‚1.408680802167848 0loss/FRt4/assignment_loss‚1.7597186985340985 1loss/SRqqt1/assignment_loss‚0.674805825556078 /loss/FRt3/detection_loss‚2.4545471559502308 /loss/FRt4/detection_loss‚2.1517408906736155 $loss/total_loss‚38.3949582353772 2loss/SRqqt4/assignment_loss‚1.1178328607689012 1loss/FBt1/assignment_loss‚0.11994456935047357 1loss/SRqqt1/detection_loss‚1.0254305579106917 0loss/SRqqt3/detection_loss‚2.889626408261912 /loss/FRt2/assignment_loss‚2.087867702756609 1loss/FBt2/assignment_loss‚0.09538414665231747 /loss/FRt1/detection_loss‚1.6416907866247172 /loss/FBt3/detection_loss‚2.4559185236692427 % _timestamp‚1.7884463530439522e+09 1loss/FBt3/assignment_loss‚0.08599121242004912 /loss/FBt1/detec68‹Btion_loss‚0.5303817225463126 trainer/global_step‚10349Ó¢_s[”…'‚(:‰ œæÔ Ý‚É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•<܃-†'‚(:¢ œæÔÀœÓÌ disk.rbd3.in‚0% disk.nvme0c0n1.out‚74672.29296875 cpu‚25.322337176538753 disk.rbd2.out‚638.2802734375 disk.sda.out‚ 320.9140625* proc.memory.percent‚16.635799407958984 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 1764.65625 network.sent‚ 2.6075155e+07) disk./.usagePercent‚74.38155696736997% disk./.usageGB‚1108.3389511108398 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 320.9140625 disk.dm-0.out‚74672.29296875$ memory_percent‚88.58098983764648! proc.memory.rssMB‚ 2725.609375 network.recv‚ 3.204116e+06 disk.rbd1.out‚ 4.35546875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1764.65625 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 320.9140625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1764.65625 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.drbd1777.in‚0# disk.nvme0n1.out‚74672.29296875' proc.memory.availableMB‚ 1870.890625 disk.rbd2.in‚0Ìþ ¡Ÿ‡'‚(:” «æÔì‚ûÈ gpu.0.gpu‚63.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø\Èeˆ'‚(:‘ «æÔ±·”Ë disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚74672.75) disk./.usagePercent‚74.38155824740764 disk.rbd1.out‚ 4.35546875 disk.rbd0.out‚0 cpu‚25.321884263609977 disk.drbd1777.in‚0 disk.rbd2.out‚638.3740234375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 321.44140625 disk.nvme0c0n1.out‚74672.75$ memory_percent‚88.58070373535156 disk.rbd2.in‚0% disk./.usageGB‚1108.3389701843262 disk.nvme0n1.in‚ 1764.65625 disk.nvme0c0n1.in‚ 1764.65625* proc.memory.percent‚16.635799407958984 disk.drbd1777.out‚0 disk.nvme0n1.out‚74672.75 network.sent‚ 2.6184421e+07 network.recv‚ 3.210577e+06 disk.dm-1.out‚ 321.44140625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1764.65625% proc.memory.availableMB‚ 1870.9375! proc.memory.rssMB‚ 2725.609375 disk.sda.out‚ 321.44140625 disk.dm-1.in‚0.5625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0.5625 proc.cpu.threads‚17ˆã¼72‰'- _wandbruntime‚8831  _runtime‚8831ŒŽlŒÉŠ'à _step‚852 ' lr-AdamW/pg1‚2.2637725869526737e-05 ' lr-AdamW/pg2‚2.2637725869526737e-05 trainer/global_step‚10399 $ _timestamp‚1.788446391396012e+09 _runtime‚8831.440694444>)Ȃ΋'È trainer/global_step‚10399 $ _timestamp‚1.788446391396012e+09 _runtime‚8831.440694444 _step‚852 ' lr-AdamW/pg1‚2.2637725869526737e-05 ' lr-AdamW/pg2‚2.2637725869526737e-05Ôzè2Œ'- _wandbruntime‚8831  _runtime‚8831ÖÀ8ï('™P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19296819280896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19264828459988 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1179082234544128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26422659158528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19296819280896 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1018552642045440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26422707652096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153552533 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1159782815053824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚992129934393344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 % _timestamp‚1.7884463913967314e+09 _runtime‚8831.441588549 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1179079607943168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99637452 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚853 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19299419490304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193996661 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72081702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚992129906258944 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1159782788662272 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1018552565417472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1179079607943168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549194 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193976627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171718688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19296819280896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19264828459988 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1179082234544128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26422659158528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19296819280896 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1018552642045440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26422707652096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153552533 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1159782815053824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚992129934393344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 % _timestamp‚1.7884463913967314e+09 _runtime‚8831.441588549 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1179079607943168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99637452 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 _step‚853 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19299419490304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193996661 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72081702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚992129906258944 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1159782788662272 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1018552565417472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1179079607943168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549194 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193976627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171718688 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Õ.”2'- _wandbruntime‚8832  _runtime‚8832+ñw!'£N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347561624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26425181884416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171734585 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193994861 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1160635425139104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194014921 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1179189379977216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _runtime‚8832.180165473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194014921 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚992222294312448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153566763 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26425230364672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99646645 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19298651609600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153566769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99646648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19301251832320 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19298651609600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1141368767336448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153566763 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171735032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 _step‚854 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153566769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72088384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19269256087532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19301251832320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72087940 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1018647552811520 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1018647476196864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚992222322446848 GADaÄÀ²eviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19266657802656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1159890754759168 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1159890754759168 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1159890728367616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1160638048851948 % _timestamp‚1.7884463921349325e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1159890754759168 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1159890754759168 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1159890728367616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1160638048851948 % _timestamp‚1.7884463921349325e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171734585 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193994861 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1160635425139104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194014921 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1179189379977216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _runtime‚8832.180165473 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194014921 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚992222294312448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153566763 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26425230364672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99646645 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19298651609600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153566769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99646648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19301251832320 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19298651609600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1141368767336448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153566763 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171735032 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 _step‚854 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153566769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72088384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19269256087532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19301251832320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72087940 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496Öï9Ýr2’'- _wandbruntime‚8832  _runtime‚8832V'^§Y“'Ó 0loss/FRt1/assignment_loss‚1.0633972530462303 /loss/FRt2/assignment_loss‚2.109401524066925 2loss/SRqqt1/assignment_loss‚0.7283578735764306 2loss/SRqqt4/assignment_loss‚1.2246539369225502 .loss/FRt2/detection_loss‚2.301914299020962 0loss/SRqqt3/detection_loss‚2.089875336204256 /loss/FBt2/detection_loss‚2.4053704341252646 1loss/SRqqt1/detection_loss‚0.8245606776804496 0loss/SRqqt2/detection_loss‚1.739271613104003 % _timestamp‚1.7884463921369643e+09 _step‚855 /loss/FRt4/assignment_loss‚1.955919888674044 .loss/FRt3/detection_loss‚2.308116834974914 0loss/FBt2/assignment_loss‚0.0992354975185271 /loss/FBt1/detection_loss‚0.6547683700978851 /loss/FBt4/detection_loss‚2.6499403544834683 _runtime‚8832.181524595 1loss/SRqqt4/detection_loss‚1.7532113802486233 %loss/total_loss‚35.62451702816966 0loss/SRqqt2/assignment_loss‚1.07211491169541 2loss/FBt3/assignment_loss‚0.008503649166827514 1loss/FBt4/assignment_loss‚0.04732028508026685 .loss/FBt3/detection_loss‚2.274434734474529 trainer/global_step‚10399 0loss/FRt3/assignment_loss‚3.0194439240864344 2loss/SRqqt3/assignment_loss‚1.4415822178125381 1loss/FBt1/assignment_loss‚0.15390545783070872 /loss/FRt1/detection_loss‚1.4803149251609433 -loss/FRt4/detection_loss‚2.21890164911747 epoch‚81‡É^”'Ø 1loss/SRqqt4/detection_loss‚1.7532113802486233 %loss/total_loss‚35.62451702816966 0loss/SRqqt2/assignment_loss‚1.07211491169541 2loss/FBt3/assignment_loss‚0.008503649166827514 1loss/FBt4/assignment_loss‚0.04732028508026685 .loss/FBt3/detection_loss‚2.274434734474529 trainer/global_step‚10399 0loss/FRt3/assignment_loss‚3.0194439240864344 2loss/SRqqt3/assignment_loss‚1.4415822178125381 1loss/FBt1/assignment_loss‚0.15390545783070872 /loss/FRt1/detection_loss‚1.4803149251609433 -loss/FRt4/detection_loss‚2.21890164911747 epoch‚8 0loss/FRt1/assignment_loss‚1.0633972530462303 /loss/FRt2/assignment_loss‚2.109401524066925 2loss/SRqqt1/assignment_loss‚0.7283578735764306 2loss/SRqqt4/assignment_loss‚1.2246539369225502 .loss/FRt2/detection_loss‚2.301914299020962 0loss/SRqqt3/detection_loss‚2.089875336204256 /loss/FBt2/detection_loss‚2.4053704341252646 1loss/SRqqt1/detection_loss‚0.8245606776804496 0loss/SRqqt2/detection_loss‚1.739271613104003 % _timestamp‚1.7884463921369643e+09 _step‚855 /loss/FRt4/assignment_loss‚1.955919888674044 .loss/FRt3/detection_loss‚2.308116834974914 0loss/FBt2/assignment_loss‚0.0992354975185271 /loss/FBt1/detection_loss‚0.6547683700978851 /loss/FBt4/detection_loss‚2.6499403544834683 _runtime‚8832.181524595×èà¿ž•'‚(:“ ºæÔ«õ‹É gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚264.557' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.91140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿLÂÌ0–'‚(:¥ ºæÔá’–Ë disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 1764.65625! disk.dm-0.out‚74771.314453125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚639.9482421875 disk.nvme0n1.in‚ 1764.65625& disk.nvme0c0n1.out‚74771.314453125 disk.dm-0.in‚ 1764.65625 disk.sda.out‚ 322.20703125 disk.sda1.in‚0 network.sent‚ 2.6189662e+07 disk.rbd0.out‚0 network.recv‚ 3.214432e+06 disk.sda3.in‚0.5625% disk./.usageGB‚1108.3381423950195 disk.sda3.out‚ 322.20703125 disk.rbd1.out‚ 4.35546875 disk.sda1.out‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 1870.6875) disk./.usagePercent‚74.38150269377272 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 322.20703125 disk.drbd1777.out‚0 cpu‚25.32142396751127$ disk.nvme0n1.out‚74771.314453125 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0* proc.memory.percent‚16.635799407958984 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.in‚0.5625$ memory_percent‚88.58222961425781! proc.memory.rssMB‚ 2725.609375€>ßüž—'‚(:“ ÉæÔŽ¦šÉ gpu.0.gpu‚93.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.888' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.77759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06&$8˜'‚(:­ ÉæÔÝ óÊ disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 1764.65625 disk.dm-1.out‚ 322.81640625 disk.dm-0.in‚ 1764.65625 network.sent‚ 2.6298771e+07% disk./.usageGB‚1108.3380889892578 disk.rbd2.in‚0 disk.rbd2.out‚640.0263671875 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚74771.74560546875 network.recv‚ 3.220637e+06& disk.nvme0n1.out‚74771.74560546875 disk.sda3.in‚0.5625# memory_percent‚88.5807991027832' proc.memory.availableMB‚ 1870.921875! proc.memory.rssMB‚ 2725.609375 disk.drbd1777.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚74.38149910966723 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 1764.65625( disk.nvme0c0n1.out‚74771.74560546875* proc.memory.percent‚16.635799407958984 disk.sda.out‚ 322.81640625 disk.sda2.out‚0 disk.sda3.out‚ 322.81640625 disk.dm-1.in‚0.5625 cpu‚25.320938786892622Ãrº<”™'‚(:‰ ØæÔƒã©É gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s+`:š'‚(:¯ ØæÔúÈ Ë disk.rbd2.out‚640.0263671875 disk.nvme0n1.in‚ 1764.65625 disk.sda3.in‚0.5625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚74869.95556640625 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 1764.65625 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 323.33984375! proc.memory.rssMB‚ 2725.609375 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.35546875) proc.memory.availableMB‚ 1870.89453125 network.sent‚ 2.6303656e+07 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚74869.95556640625* proc.memory.percent‚16.635799407958984 network.recv‚ 3.224428e+06) disk./.usagePercent‚74.38148733332066 disk.rbd0.in‚0 cpu‚25.32050774860613% disk./.usageGB‚1108.3379135131836 disk.drbd1777.out‚0$ memory_percent‚88.58096599578857# disk.dm-0.out‚74869.95556640625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 323.33984375 disk.sda3.out‚ 323.33984375 disk.dm-0.in‚ 1764.65625 proc.cpu.threads‚17 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.rbd3.out‚0ý­72›'- _wandbruntime‚8870  _runtime‚8870:g§oÈœ' ' lr-AdamW/pg1‚2.1343582314186374e-05 ' lr-AdamW/pg2‚2.1343582314186374e-05 trainer/global_step‚10449 # _timestamp‚1.78844643079749e+09 _runtime‚8870.842132267 _step‚856¯Ó -Í'Ç ' lr-AdamW/pg1‚2.1343582314186374e-05 ' lr-AdamW/pg2‚2.1343582314186374e-05 trainer/global_step‚10449 # _timestamp‚1.78844643079749e+09 _runtime‚8870.842132267 _step‚856Øa4œÑ2ž'- _wandbruntime‚8870  _runtime‚8870-ŸH!(Ÿ'›P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1165179800321024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26548478904320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19358907469264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19356309184596 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚8870.843664394 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1184570850660352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1184568224059392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚996749337068544 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172516418 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19391050339328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1023297815972864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 % _timestamp‚1.7884464307984462e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194891679 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1023297739344896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194891679 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194911713 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1165179800321024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19391050339328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154264333 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176046 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1165935794060372 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194891679 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1023297739344896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194891679 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194911713 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1165179800321024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19391050339328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154264333 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176046 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1165935794060372 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1165179800321024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26548478904320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19358907469264 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19356309184596 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 _runtime‚8870.843664394 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1184570850660352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1184568224059392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚996749337068544 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172516418 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19391050339328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1023297815972864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 % _timestamp‚1.7884464307984462e+09 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1165287740026368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚996841725122048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19390282458624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194929973 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1023392712751104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100106445 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚858 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194909913 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19390282458624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1184677996093440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100106448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1146685851478016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1165287713634816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1184680622707712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26550987629056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72425895 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1146685826050048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208542 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154278569 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _runtime‚8871.579927475 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1166046588290156 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172531906 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚996841696987648K#ÈÕ0'£'ªN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884464315348585e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19392882681344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154278563 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1165287740026368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚996841725122048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19390282458624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194929973 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1023392712751104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100106445 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚858 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194909913 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19390282458624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1184677996093440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100106448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1146685851478016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1165287713634816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1184680622707712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26550987629056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349188476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72425895 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1146685826050048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208542 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154278569 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _runtime‚8871.579927475 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1166046588290156 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172531906 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚996841696987648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1184680622707712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19360736812140ÚhkÕ2¤'- _wandbruntime‚8871  _runtime‚8871•ý³>[¥'Õ 0loss/FRt1/assignment_loss‚1.1218355880542235 /loss/FBt4/detection_loss‚2.6737872302532195 epoch‚8 2loss/SRqqt3/assignment_loss‚1.5034326618680587 .loss/FRt4/detection_loss‚2.560184457235866 0loss/SRqqt4/detection_loss‚2.104688275436109 /loss/FBt1/detection_loss‚0.5689757737361462 _step‚859 1loss/FBt4/assignment_loss‚0.08073737529775826 % _timestamp‚1.7884464315368042e+09 /loss/FRt2/assignment_loss‚2.188585214695688 1loss/FBt3/assignment_loss‚0.15538697184880826 trainer/global_step‚10449 /loss/FRt3/assignment_loss‚2.831846881657839 /loss/FRt2/detection_loss‚1.9384729235859242 1loss/SRqqt1/detection_loss‚0.8850529664947141 1loss/SRqqt3/detection_loss‚2.1159075636130114 %loss/total_loss‚36.97050300858045 /loss/FRt4/assignment_loss‚2.331245536605517 1loss/FBt1/assignment_loss‚0.06378854274826705 .loss/FRt1/detection_loss‚1.491000134392049 /loss/FRt3/detection_loss‚2.6581201903936744 /loss/FBt3/detection_loss‚2.4310150345166526 2loss/SRqqt1/assignment_loss‚0.9889352856985023 1loss/SRqqt4/assignment_loss‚1.173229534919797 1loss/FBt2/assignment_loss‚0.12058696584691378 .loss/FBt2/detection_loss‚2.392685727639632 _runtime‚8871.581367859 1loss/SRqqt2/assignment_loss‚1.101299142914026 1loss/SRqqt2/detection_loss‚1.4897030291280575“wcî`¦'Ú 1loss/SRqqt2/assignment_loss‚1.101299142914026 1loss/SRqqt2/detection_loss‚1.4897030291280575 0loss/FRt1/assignment_loss‚1.1218355880542235 /loss/FBt4/detection_loss‚2.6737872302532195 epoch‚8 2loss/SRqqt3/assignment_loss‚1.5034326618680587 .loss/FRt4/detection_loss‚2.560184457235866 0loss/SRqqt4/detection_loss‚2.104688275436109 /loss/FBt1/detection_loss‚0.5689757737361462 _step‚859 1loss/FBt4/assignment_loss‚0.08073737529775826 % _timestamp‚1.7884464315368042e+09 /loss/FRt2/assignment_loss‚2.188585214695688 1loss/FBt3/assignment_loss‚0.15538697184880826 trainer/global_step‚10449 /loss/FRt3/assignment_loss‚2.831846881657839 /loss/FRt2/detection_loss‚1.9384729235859242 1loss/SRqqt1/detection_loss‚0.8850529664947141 1loss/SRqqt3/detection_loss‚2.1159075636130114 %loss/total_loss‚36.97050300858045 /loss/FRt4/assignment_loss‚2.331245536605517 1loss/FBt1/assignment_loss‚0.06378854274826705 .loss/FRt1/detection_loss‚1.491000134392049 /loss/FRt3/detection_loss‚2.6581201903936744 /loss/FBt3/detection_loss‚2.4310150345166526 2loss/SRqqt1/assignment_loss‚0.9889352856985023 1loss/SRqqt4/assignment_loss‚1.173229534919797 1loss/FBt2/assignment_loss‚0.12058696584691378 .loss/FBt2/detection_loss‚2.392685727639632 _runtime‚8871.581367859Û쟃£Ÿ§'‚(:” çæÔ™¨°É gpu.0.gpu‚69.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±½•O2¨'‚(:§ çæÔŽ˜Ë disk.sda3.out‚ 323.859375 disk.dm-0.in‚ 1764.65625$ memory_percent‚88.58144283294678 cpu‚25.32002448147884 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚74870.2802734375) proc.memory.availableMB‚ 1870.81640625 disk.sda3.in‚0.5625 disk.dm-1.out‚ 323.859375 network.recv‚ 3.228739e+06% disk./.usageGB‚1108.3379135131836 disk.nvme0n1.in‚ 1764.65625 disk.nvme0c0n1.in‚ 1764.65625" disk.dm-0.out‚74870.2802734375 disk.sda.out‚ 323.859375* proc.memory.percent‚16.635799407958984 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625! proc.memory.rssMB‚ 2725.609375 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.35546875! disk.rbd2.out‚640.12841796875 network.sent‚ 2.6309898e+07) disk./.usagePercent‚74.38148733332066 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚74870.2802734375 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0.5625 disk.sda2.out‚0 disk.drbd1777.in‚0½ž{ð”©'‚(:‰ öæÔÊÿ•É gpu.0.gpu‚93.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚244.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ªeâª'‚(:¬ öæÔåš‹Ì# disk.dm-0.out‚74969.84326171875 disk.sda1.out‚0 disk.sda3.out‚ 324.5234375* proc.memory.percent‚16.635799407958984) disk./.usagePercent‚74.38126460676587 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 324.5234375! disk.rbd2.out‚640.12841796875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1764.65625 disk.drbd1777.in‚0 disk.rbd0.in‚0$ memory_percent‚88.58113288879395 disk.rbd1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.6418202e+07 disk.drbd1777.out‚0 disk.sda.out‚ 324.5234375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 1764.65625! proc.memory.rssMB‚ 2725.609375 cpu‚25.31956526767733ê’¯ÀU disk.sda.in‚0.5625 network.recv‚ 3.234693e+06 disk.dm-0.in‚ 1764.65625 disk.rbd2.in‚0 disk.sda3.in‚0.5625( disk.nvme0c0n1.out‚74969.84326171875( proc.memory.availableMB‚ 1870.8671875% disk./.usageGB‚1108.3345947265625 disk.rbd1.out‚ 4.35546875 disk.sda2.in‚0& disk.nvme0n1.out‚74969.84326171875 È.,”«'‚(:‰ …ŽæÔþªùÈ gpu.0.gpu‚94.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚260.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I`ø³6¬'‚(:« …ŽæÔæ«…Ë disk.dm-1.in‚0.5625 disk.sda2.in‚0 disk.nvme0n1.in‚ 1764.6953125! proc.memory.rssMB‚ 2725.609375 network.recv‚ 3.239157e+06) disk./.usagePercent‚74.38126076665286% disk./.usageGB‚1108.3345375061035 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.635799407958984 disk.rbd1.out‚ 4.50390625! disk.rbd2.out‚641.84326171875 disk.dm-1.out‚ 325.484375 proc.cpu.threads‚17 disk.dm-0.in‚ 1764.6953125 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 325.484375 disk.rbd3.out‚0$ disk.nvme0n1.out‚74970.943359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 325.484375" disk.nvme0c0n1.in‚ 1764.6953125& disk.nvme0c0n1.out‚74970.943359375 disk.rbd1.in‚0$ memory_percent‚88.58129978179932) proc.memory.availableMB‚ 1870.83984375 cpu‚25.319146912753236 network.sent‚ 2.6424477e+07! disk.dm-0.out‚74970.943359375d9_2­'- _wandbruntime‚8910  _runtime‚8910‰ÔúGÉ®'à _runtime‚ 8910.07200246 _step‚860 ' lr-AdamW/pg1‚2.0085369466472602e-05 ' lr-AdamW/pg2‚2.0085369466472602e-05 trainer/global_step‚10499 % _timestamp‚1.7884464700272977e+09ï…o8ί'È ' lr-AdamW/pg2‚2.0085369466472602e-05 trainer/global_step‚10499 % _timestamp‚1.7884464700272977e+09 _runtime‚ 8910.07200246 _step‚860 ' lr-AdamW/pg1‚2.0085369466472602e-05Ü›¥Ê¹2°'- _wandbruntime‚8910  _runtime‚8910ÅA¾š"(±'œP [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19480080978944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1190059466776576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100557052 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028042743981568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72755820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1001368739743744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154976127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195806731 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173312872 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195806731 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 _runtime‚8910.072943123 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1170576759196672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1171344333498580 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350782858 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195826765 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1151896569017344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1190059466776576 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100557049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1170576759196672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚861 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195826765 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 $ _timestamp‚1.788446470028071e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19482681188352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26674080865792 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1190056840175616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400á°í2'(²'¡P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19482681188352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26674080865792 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1190056840175616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19480080978944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1190059466776576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100557052 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028042743981568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72755820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1001368739743744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154976127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195806731 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173312872 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195806731 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 _runtime‚8910.072943123 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1170576759196672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1171344333498580 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350782858 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195826765 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1151896569017344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1190059466776576 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100557049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1170576759196672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚861 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195826765 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 $ _timestamp‚1.788446470028071e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552Ý?Ù3á2³'- _wandbruntime‚8911  _runtime‚8911?r³Ð&´'§N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1190169238823936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72762445 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26676592810496 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1170684725293568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173328239 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1171452504015520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170684698902016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195824965 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100566248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 % _timestamp‚1.7884464710074918e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170684725293568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154990363 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19481913307648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19481913307648 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195845025 _step‚862 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72761994 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1001461099662848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 _runtime‚8911.052840305 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26676544330240 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154990369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195824965 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1001461127797248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154990369 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1028137643993088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19484513530368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028137720607744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100566245 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1190166612209664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195845025 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19484513530368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350815328 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173328693 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10499 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 _runtime‚8911.052840305 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26676544330240 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154990369 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195824965 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1001461127797248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154990369 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1028137643993088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19484513530368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028137720607744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100566245 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1190166612209664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195845025 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19484513530368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350815328 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173328693 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10499 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173328239 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1171452504015520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170684698902016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195824965 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100566248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 % _timestamp‚1.7884464710074918e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170684725293568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154990363 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Þ£@˜>2¶'- _wandbruntime‚8911  _runtime‚8911#š\I_·'Ù /loss/FBt3/detection_loss‚2.3872331976890564 /loss/FRt2/assignment_loss‚2.015926206750529 _runtime‚8911.053897798 0loss/FRt4/assignment_loss‚1.9759512509939805 2loss/FBt4/assignment_loss‚0.027998256824503187 /loss/FRt2/detection_loss‚2.2541387961911306 1loss/SRqqt4/detection_loss‚1.7744314439119178 %loss/total_loss‚35.66750717864167 2loss/SRqqt4/assignment_loss‚1.1942506757505396 0loss/FBt3/assignment_loss‚0.1987689900173185 /loss/FBt4/detection_loss‚3.0126487463712692 trainer/global_step‚10499 % _timestamp‚1.7884464710093412e+09 _step‚863 1loss/FBt2/assignment_loss‚0.22519091536378255 .loss/FRt1/detection_loss‚1.575955263784398 1loss/SRqqt2/detection_loss‚1.4971428963801134 /loss/FRt3/detection_loss‚2.7263348853318043 .loss/FRt4/detection_loss‚2.201150138001397 1loss/SRqqt1/detection_loss‚0.9035177850492404 0loss/SRqqt3/detection_loss‚2.795426337807267 0loss/FRt1/assignment_loss‚0.7782874755237413 0loss/FRt3/assignment_loss‚2.8010331551233927 2loss/SRqqt1/assignment_loss‚0.7911872806578455 2loss/SRqqt2/assignment_loss‚1.0957524765930746 1loss/FBt1/assignment_loss‚0.11719576786302326 epoch‚8 2loss/SRqqt3/assignment_loss‚1.0502629777485575 /loss/FBt1/detection_loss‚0.5863484541061208 .loss/FBt2/detection_loss‚1.681373804807663…xõ‹d¸'Þ 1loss/FBt2/assignment_loss‚0.22519091536378255 .loss/FRt1/detection_loss‚1.575955263784398 1loss/SRqqt2/detection_loss‚1.4971428963801134 /loss/FRt3/detection_loss‚2.7263348853318043 .loss/FRt4/detection_loss‚2.201150138001397 1loss/SRqqt1/detection_loss‚0.9035177850492404 0loss/SRqqt3/detection_loss‚2.795426337807267 0loss/FRt1/assignment_loss‚0.7782874755237413 0loss/FRt3/assignment_loss‚2.8010331551233927 2loss/SRqqt1/assignment_loss‚0.7911872806578455 2loss/SRqqt2/assignment_loss‚1.0957524765930746 1loss/FBt1/assignment_loss‚0.11719576786302326 epoch‚8 2loss/SRqqt3/assignment_loss‚1.0502629777485575 /loss/FBt1/detection_loss‚0.5863484541061208 .loss/FBt2/detection_loss‚1.681373804807663 /loss/FBt3/detection_loss‚2.3872331976890564 /loss/FRt2/assignment_loss‚2.015926206750529 _runtime‚8911.053897798 0loss/FRt4/assignment_loss‚1.9759512509939805 2loss/FBt4/assignment_loss‚0.027998256824503187 /loss/FRt2/detection_loss‚2.2541387961911306 1loss/SRqqt4/detection_loss‚1.7744314439119178 %loss/total_loss‚35.66750717864167 2loss/SRqqt4/assignment_loss‚1.1942506757505396 0loss/FBt3/assignment_loss‚0.1987689900173185 /loss/FBt4/detection_loss‚3.0126487463712692 trainer/global_step‚10499 % _timestamp‚1.7884464710093412e+09 _step‚863ß–P韹'‚(:” ”ŽæÔöŸÉ gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.962799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…z~ͺ'‚(:‰ ”ŽæÔ–·Ë disk.sda1.in‚0 disk.dm-1.out‚ 325.796875 disk.sda.out‚ 325.796875 disk.nvme0n1.in‚ 1767.171875 cpu‚25.318665705010066 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2725.609375 proc.cpu.threads‚17* proc.memory.percent‚16.635799407958984 disk.nvme0n1.out‚74974) disk./.usagePercent‚74.38129276759464 disk.sda.in‚0.5625 disk.sda3.out‚ 325.796875 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 1767.171875! disk.rbd2.out‚641.84326171875 network.recv‚ 3.243613e+06 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.50390625$ memory_percent‚88.58304023742676( proc.memory.availableMB‚ 1870.5546875 disk.rbd1.in‚0 network.sent‚ 2.6430229e+07% disk./.usageGB‚1108.3350143432617 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.nvme0c0n1.out‚74974 disk.dm-0.in‚ 1767.171875 disk.dm-0.out‚74974 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0}8Ñò”»'‚(:‰ £ŽæÔê™—É gpu.0.gpu‚82.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚249.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-9Öd/¼'‚(:¤ £ŽæÔàö„Ë disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 1767.65625 disk.dm-1.out‚ 326.78125! disk.rbd2.out‚641.92138671875 disk.sda.in‚0.5625$ memory_percent‚88.58151435852051 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2725.609375 disk.nvme0n1.in‚ 1767.65625* proc.memory.percent‚16.635799407958984 proc.cpu.threads‚17% disk./.usageGB‚1108.3353652954102 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.dm-0.in‚ 1767.65625 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 4.5361328125( proc.memory.availableMB‚ 1870.8046875 network.sent‚ 2.6539664e+07 network.recv‚ 3.251005e+06" disk.dm-0.out‚75072.0556640625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚75072.0556640625 disk.drbd1777.out‚0 disk.sda.out‚ 326.78125 disk.rbd3.out‚0( disk./.usagePercent‚74.3813163202878 disk.sda3.out‚ 326.78125' disk.nvme0c0n1.out‚75072.0556640625 cpu‚25.31824650304246)w²¶2½'- _wandbruntime‚8949  _runtime‚8949’Å°+ʾ'Ä ' lr-AdamW/pg1‚1.8863349536057052e-05 ' lr-AdamW/pg2‚1.8863349536057052e-05 trainer/global_step‚10549 % _timestamp‚1.7884465094740143e+09 _runtime‚8949.518760195 _step‚864–žÏ¿'É ' lr-AdamW/pg1‚1.8863349536057052e-05 ' lr-AdamW/pg2‚1.8863349536057052e-05 trainer/global_step‚10549 % _timestamp‚1.7884465094740143e+09 _runtime‚8949.518760195 _step‚864àZš#µ2À'- _wandbruntime‚8949  _runtime‚8949r2C#(Á'P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1175973744463872 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1157213627730944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195545456291840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1175973770855424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196721783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174107528 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1032787673825280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1176755496649424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196721783 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101016852 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196741817 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195548082892800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19539270633812 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005988114284544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26799559540736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19571711827968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73090679 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26799608034304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196741817 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429750 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 _step‚865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 % _timestamp‚1.7884465094748287e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19571711827968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19574312037376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1176752872936788 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195545456291840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73091140 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005988142418944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155687927 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155687933 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1157213602302976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1175973744463872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174107992 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1032787750453248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0’K'áËÂ'¢P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195545456291840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73091140 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005988142418944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155687927 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155687933 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1157213602302976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1175973744463872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174107992 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1032787750453248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1175973744463872 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1157213627730944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195545456291840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1175973770855424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196721783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174107528 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1032787673825280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1176755496649424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196721783 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101016852 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196741817 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352409710 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195548082892800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19539270633812 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005988114284544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26799559540736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19571711827968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73090679 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26799608034304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196741817 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429750 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 _step‚865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 % _timestamp‚1.7884465094748287e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19571711827968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19574312037376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1176752872936788 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776á¡ ø2Ã'- _wandbruntime‚8950  _runtime‚8950ú@ãâ-'Ä'§N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155702169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196740017 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155702163 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174123966 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26802117125120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19573544156672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174123506 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19576144379392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1157319943477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196760077 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _step‚866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1195655228325888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1176081684169216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1195657854940160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19541099976480 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101026048 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101026045 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196740017 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1006080502338048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1157319968905216 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155702169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73097461 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26802068644864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 % _timestamp‚1.7884465102201707e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 _runtime‚8950.265308246 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1176081710560768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196760077 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1176863667166572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1006080530472448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1176861043453728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73097918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1176081710560768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚10549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1176081684169216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174123966 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26802117125120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19573544156672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174123506 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19576144379392 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1157319943477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196760077 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _step‚866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1195655228325888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1176081684169216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1195657854940160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19541099976480 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101026048 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101026045 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196740017 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1006080502338048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1157319968905216 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155702169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73097461 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26802068644864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 % _timestamp‚1.7884465102201707e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 _runtime‚8950.265308246 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1176081710560768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196760077 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1176863667166572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1006080530472448 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1176861043453728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73097918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1176081710560768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0â=“°'2Æ'- _wandbruntime‚8950  _runtime‚8950¼JWÇ'Ñ 2loss/SRqqt3/assignment_loss‚1.4463996708844646 0loss/SRqqt4/detection_loss‚1.502512264863721 0loss/FBt2/assignment_loss‚0.6374183219547073 .loss/FRt3/detection_loss‚2.505818758607451 0loss/FRt2/assignment_loss‚2.6657754606670805 1loss/FBt3/assignment_loss‚0.10199744060519152 .loss/FRt4/detection_loss‚2.163379117846489 2loss/SRqqt4/assignment_loss‚1.3215046969874362 0loss/FBt4/assignment_loss‚0.6034785411470693 /loss/FBt3/detection_loss‚2.3842437148094175 _step‚867 .loss/FRt2/detection_loss‚2.332521856824557 0loss/FBt1/assignment_loss‚0.1242656819535391 /loss/FRt1/detection_loss‚1.8360445892963655 .loss/FBt1/detection_loss‚0.609303621504503 /loss/FBt2/detection_loss‚1.9556834002335866 2loss/SRqqt2/assignment_loss‚1.2844527626738829 %loss/total_loss‚37.27360619368399 0loss/FRt1/assignment_loss‚0.8150106790738229 /loss/FRt4/assignment_loss‚2.272895174947652 1loss/SRqqt3/detection_loss‚2.1306543883049125 epoch‚8 2loss/SRqqt1/assignment_loss‚0.8430091979612103 1loss/SRqqt1/detection_loss‚0.8935126703889931 0loss/SRqqt2/detection_loss‚1.691630523870973 .loss/FBt4/detection_loss‚2.282505933727537 % _timestamp‚1.7884465102223568e+09 _runtime‚8950.266940653 trainer/global_step‚10549 .loss/FRt3/assignment_loss‚2.86958772454943—)1\È'Ö 1loss/SRqqt1/detection_loss‚0.8935126703889931 0loss/SRqqt2/detection_loss‚1.691630523870973 .loss/FBt4/detection_loss‚2.282505933727537 % _timestamp‚1.7884465102223568e+09 _runtime‚8950.266940653 trainer/global_step‚10549 .loss/FRt3/assignment_loss‚2.86958772454943 2loss/SRqqt3/assignment_loss‚1.4463996708844646 0loss/SRqqt4/detection_loss‚1.502512264863721 0loss/FBt2/assignment_loss‚0.6374183219547073 .loss/FRt3/detection_loss‚2.505818758607451 0loss/FRt2/assignment_loss‚2.6657754606670805 1loss/FBt3/assignment_loss‚0.10199744060519152 .loss/FRt4/detection_loss‚2.163379117846489 2loss/SRqqt4/assignment_loss‚1.3215046969874362 0loss/FBt4/assignment_loss‚0.6034785411470693 /loss/FBt3/detection_loss‚2.3842437148094175 _step‚867 .loss/FRt2/detection_loss‚2.332521856824557 0loss/FBt1/assignment_loss‚0.1242656819535391 /loss/FRt1/detection_loss‚1.8360445892963655 .loss/FBt1/detection_loss‚0.609303621504503 /loss/FBt2/detection_loss‚1.9556834002335866 2loss/SRqqt2/assignment_loss‚1.2844527626738829 %loss/total_loss‚37.27360619368399 0loss/FRt1/assignment_loss‚0.8150106790738229 /loss/FRt4/assignment_loss‚2.272895174947652 1loss/SRqqt3/detection_loss‚2.1306543883049125 epoch‚8 2loss/SRqqt1/assignment_loss‚0.8430091979612103ãÁàc’žÉ'‚(:“ ²ŽæÔõ¦ôÈ gpu.0.gpu‚94.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚255.638' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.12760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß®­ƒEÊ'‚(:º ²ŽæÔ–ŃË disk.drbd1777.out‚0 disk.rbd1.out‚ 4.5361328125! disk.rbd2.out‚641.92138671875( proc.memory.availableMB‚ 1870.5234375* proc.memory.percent‚16.635823249816895 disk.rbd3.in‚0( disk.nvme0c0n1.out‚75073.73486328125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 1768.3359375 disk.dm-1.out‚ 327.11328125 disk.drbd1777.in‚0 cpu‚25.317770731730405 disk.rbd0.out‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.out‚ 327.11328125# disk.dm-0.out‚75073.73486328125 disk.rbd1.in‚0$ memory_percent‚88.58323097229004 network.recv‚ 3.254896e+06 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚75073.73486328125 disk.sda3.in‚0.5625 disk.rbd2.in‚0 network.sent‚ 2.6544847e+07) disk./.usagePercent‚74.38121340525903% disk./.usageGB‚1108.3338317871094 disk.dm-1.in‚0.5625 disk.sda.out‚ 327.11328125# proc.memory.rssMB‚ 2725.61328125 disk.nvme0n1.in‚ 1768.3359375 disk.dm-0.in‚ 1768.3359375:f×æ”Ë'‚(:‰ ÁŽæÔÓŒ¦É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚271.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ºUã;Ì'‚(:° ÁŽæÔ‹ñ£Ë network.recv‚ 3.261178e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 4.59912109375 disk.rbd2.out‚ 642.0078125 disk.sda2.out‚0 disk.dm-0.in‚ 1769.25390625 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625 disk.rbd3.in‚0 disk.rbd0.in‚0# memory_percent‚88.5817289352417# proc.memory.rssMB‚ 2725.61328125 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0$ disk./.usageGB‚1108.336036682129 disk.drbd1777.in‚0 cpu‚25.317322794413244 proc.cpu.threads‚17 disk.dm-1.out‚ 327.96875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚75171.20751953125( disk.nvme0c0n1.out‚75171.20751953125* proc.memory.percent‚16.635823249816895) disk./.usagePercent‚74.38136137761381# disk.dm-0.out‚75171.20751953125 disk.sda1.in‚0 disk.sda3.out‚ 327.96875 network.sent‚ 2.6654013e+07 disk.sda.out‚ 327.96875! disk.nvme0n1.in‚ 1769.25390625# disk.nvme0c0n1.in‚ 1769.25390625) proc.memory.availableMB‚ 1870.76953125.µº”Í'‚(:‰ ÐŽæÔ¥ÕåÈ gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚246.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.3462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³k˜=Î'‚(:² ÐŽæÔîûçÌ disk.sda1.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1769.25390625# disk.nvme0c0n1.in‚ 1769.25390625 disk.dm-1.out‚ 328.28125 network.sent‚ 2.6660003e+07 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.59912109375 disk.sda.in‚0.5625& disk.nvme0n1.out‚75172.13525390625 disk.rbd2.in‚0 disk.sda.out‚ 328.28125% disk./.usageGB‚1108.3359413146973 disk.drbd1777.out‚0 disk.sda3.in‚0.5625# disk.dm-0.out‚75172.13525390625 cpu‚25.316868966186224) disk./.usagePercent‚74.38135497742546 disk.rbd3.in‚0 disk.rbd0.in‚0# memory_percent‚88.5817289352417# proc.memory.rssMB‚ 2725.61328125* proc.memory.percent‚16.635823249816895 network.recv‚ 3.265443e+06( disk.nvme0c0n1.out‚75172.13525390625 proc.cpu.threads‚17 disk.rbd2.out‚ 642.01953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 328.28125 disk.dm-0.in‚ 1769.25390625 disk.dm-1.in‚0.5625 disk.rbd3.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1870.76953125®p‡2Ï'- _wandbruntime‚8988  _runtime‚8988ª ³ÊÐ'Ä trainer/global_step‚10599 % _timestamp‚1.7884465489461203e+09 _runtime‚8988.990816964 _step‚868 ' lr-AdamW/pg1‚1.7677777190061548e-05 ' lr-AdamW/pg2‚1.7677777190061548e-05áYÎÏÑ'É trainer/global_step‚10599 % _timestamp‚1.7884465489461203e+09 _runtime‚8988.990816964 _step‚868 ' lr-AdamW/pg1‚1.7677777190061548e-05 ' lr-AdamW/pg2‚1.7677777190061548e-05äQWƒa2Ò'- _wandbruntime‚8988  _runtime‚8988GR·v/Ó'œP YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_byäšLó&tes.all.freed‚1201034072408064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101476652 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚489 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19665942886400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19663342676992 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1037532510084608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1162530661016576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1181370756122624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1201036699009024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19663342676992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156399733 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156399727 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1182161412374996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚492 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036562 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1162530686444544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174901275 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1201036699009024 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1010607545094144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197656869 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26925045812736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174901767 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 354056602 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197636835 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26924993124864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101476649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156399727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 $ _timestamp‚1.788446548946988e+09 _runtime‚8988.991804476 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156399733 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1181370756122624 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Srç'(Ô'¡P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1162530686444544 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174901275 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1201036699009024 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1010607545094144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197656869 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚26925045812736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174901767 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 354056602 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197636835 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚26924993124864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101476649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156399727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 $ _timestamp‚1.788446548946988e+09 _runtime‚8988.991804476 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156399733 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1181370756122624 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1201034072408064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101476652 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚489 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19665942886400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19663342676992 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1037532510084608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1162530661016576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1181370756122624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1201036699009024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19663342676992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156399733 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156399727 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1182161412374996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚492 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036562åz‰i2Õ'- _wandbruntime‚8989  _runtime‚8989¥Ô[”-'Ö'§N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354089098 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156413969 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1201146471056384 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1181478669436416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1181478695827968 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101485848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 354069032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26927535880704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19667775228416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197675129 trainer/global_step‚10599 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1181478669436416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069032 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197655069 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174917221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101485845 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037627469028352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19665175005696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1162637002190848 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73431847 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174917695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1201146471056384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1182269582891936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1010699905013248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _step‚870 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156413969 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197655069 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1182272206604780 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197675129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73431376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26927483206144 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 _runtime‚8989.728477557 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156413963 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156413963 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1037627388219392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19667775228416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1010699933147648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1201143844442112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19632580701088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19635178985964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884465496831355e+093~DDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174917221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101485845 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1037627469028352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19665175005696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1162637002190848 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73431847 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174917695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SFDeviceStatsMonitor.on_train_bPÔ¬Îatch_end/inactive_split_bytes.all.current‚80808960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1201146471056384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1182269582891936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1010699905013248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _step‚870 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156413969 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197655069 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1182272206604780 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197675129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73431376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚26927483206144 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 _runtime‚8989.728477557 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156413963 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156413963 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1037627388219392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19667775228416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1010699933147648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1201143844442112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19632580701088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19635178985964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884465496831355e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354089098 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156413969 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1201146471056384 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1181478669436416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1181478695827968 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101485848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 354069032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚26927535880704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753æøiÅ2Ø'- _wandbruntime‚8989  _runtime‚8989”)¦»[Ù'Õ /loss/SRqqt3/detection_loss‚1.86334962323308 0loss/FBt1/detection_loss‚0.47077203993327343 -loss/FRt1/detection_loss‚1.71445425234735 _runtime‚8989.729914095 2loss/SRqqt1/assignment_loss‚1.0699775132235507 0loss/SRqqt2/detection_loss‚1.808128366327804 _step‚871 0loss/FRt4/assignment_loss‚2.3558986148935683 2loss/SRqqt3/assignment_loss‚1.0548053546110168 /loss/FRt3/detection_loss‚2.5122546714936527 /loss/FBt2/detection_loss‚1.8258056044578552 trainer/global_step‚10599 %loss/total_loss‚35.95822611739501 % _timestamp‚1.7884465496853476e+09 2loss/SRqqt4/assignment_loss‚1.3164190697882856 1loss/FBt1/assignment_loss‚0.02260097758285393 1loss/FBt2/assignment_loss‚0.30698791758768496 .loss/FRt2/detection_loss‚2.617283130117825 .loss/FRt4/detection_loss‚2.006459540667686 0loss/SRqqt1/detection_loss‚1.109163012555135 /loss/FRt2/assignment_loss‚2.494525931562696 /loss/FRt3/assignment_loss‚2.761233870799725 1loss/FBt3/assignment_loss‚0.14168748816674842 1loss/SRqqt4/detection_loss‚1.7027452175638504 /loss/FBt3/detection_loss‚2.4857830745833263 0loss/FRt1/assignment_loss‚0.9696818792819977 2loss/SRqqt2/assignment_loss‚1.1666765233866223 1loss/FBt4/assignment_loss‚0.13419144087108886 /loss/FBt4/detection_loss‚2.0473410023583307 epoch‚8^ã@÷`Ú'Ú 1loss/SRqqt4/detection_loss‚1.7027452175638504 /loss/FBt3/detection_loss‚2.4857830745833263 0loss/FRt1/assignment_loss‚0.9696818792819977 2loss/SRqqt2/assignment_loss‚1.1666765233866223 1loss/FBt4/assignment_loss‚0.13419144087108886 /loss/FBt4/detection_loss‚2.0473410023583307 epoch‚8 /loss/SRqqt3/detection_loss‚1.86334962323308 0loss/FBt1/detection_loss‚0.47077203993327343 -loss/FRt1/detection_loss‚1.71445425234735 _runtime‚8989.729914095 2loss/SRqqt1/assignment_loss‚1.0699775132235507 0loss/SRqqt2/detection_loss‚1.808128366327804 _step‚871 0loss/FRt4/assignment_loss‚2.3558986148935683 2loss/SRqqt3/assignment_loss‚1.0548053546110168 /loss/FRt3/detection_loss‚2.5122546714936527 /loss/FBt2/detection_loss‚1.8258056044578552 trainer/global_step‚10599 %loss/total_loss‚35.95822611739501 % _timestamp‚1.7884465496853476e+09 2loss/SRqqt4/assignment_loss‚1.3164190697882856 1loss/FBt1/assignment_loss‚0.02260097758285393 1loss/FBt2/assignment_loss‚0.30698791758768496 .loss/FRt2/detection_loss‚2.617283130117825 .loss/FRt4/detection_loss‚2.006459540667686 0loss/SRqqt1/detection_loss‚1.109163012555135 /loss/FRt2/assignment_loss‚2.494525931562696 /loss/FRt3/assignment_loss‚2.761233870799725 1loss/FBt3/assignment_loss‚0.14168748816674842ç.¨vš”Û'‚(:‰ ߎæÔ® ¸É gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚151.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½™šŠAÜ'‚(:¶ ߎæÔÌ—ÆË disk.sda.in‚0.5625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1769.25390625 disk.dm-1.out‚ 328.79296875 disk.rbd1.in‚0 network.recv‚ 3.270246e+06 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚88.58211040496826) proc.memory.availableMB‚ 1870.70703125) disk./.usagePercent‚74.38137008186997 disk.rbd2.out‚ 643.71875 disk.sda2.in‚0( disk.nvme0c0n1.out‚75172.52099609375 cpu‚25.316460489628835 disk.sda1.in‚0 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 328.79296875 disk.dm-0.in‚ 1769.25390625$ disk./.usageGB‚1108.336166381836# disk.dm-0.out‚75172.52099609375 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2725.61328125 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1769.25390625& disk.nvme0n1.out‚75172.52099609375 proc.cpu.threads‚17 network.sent‚ 2.6666227e+07 disk.sda.out‚ 328.79296875 disk.rbd0.out‚0 disk.rbd1.out‚ 4.662109375* proc.memory.percent‚16.635823249816895óÔ“Ý'‚(:ˆ îŽæÔåÁ¬É gpu.0.gpu‚54.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚224.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÜD@Þ'‚(:µ îŽæÔ”í£Ë network.recv‚ 3.276123e+06% disk./.usageGB‚1108.3360710144043 disk.sda3.out‚ 329.16796875# disk.dm-0.out‚75270.59619140625# proc.memory.rssMB‚ 2725.61328125 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚75270.59619140625# disk.nvme0c0n1.in‚ 1769.25390625* proc.memory.percent‚16.635823249816895 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 329.16796875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 1769.25390625 network.sent‚ 2.6774464e+07! disk.nvme0n1.in‚ 1769.25390625( disk.nvme0c0n1.out‚75270.59619140625 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚88.58366012573242 cpu‚25.31599238209773) disk./.usagePercent‚74.38136368168162 disk.sda.out‚ 329.16796875 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚ 643.734375 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 1870.453125 disk.rbd1.out‚ 4.662109375¬·c¦2ß'- _wandbruntime‚9028  _runtime‚9028͵ówÈà' % _timestamp‚1.7884465881788142e+09 _runtime‚9028.223473028 _step‚872 & lr-AdamW/pg1‚1.652889949998604e-05 & lr-AdamW/pg2‚1.652889949998604e-05 trainer/global_step‚10649ú)ÙxÍá'Ç _step‚872 & lr-AdamW/pg1‚1.652889949998604e-05 & lr-AdamW/pg2‚1.652889949998604e-05 trainer/global_step‚10649 % _timestamp‚1.7884465881788142e+09 _runtime‚9028.223473028èè!žª2â'- _wandbruntime‚9028  _runtime‚9028À«už#(ã'P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19754973526016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198571921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198571921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19722232083028 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 % _timestamp‚1.7884465881796317e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1206525315125248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1042277702071808 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157111533 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1015226947769344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27050782436864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19754973526016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198551887 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1187572575525840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19757573735424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157111533 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157111527 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1206522688524288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1015226919634944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1187569951813204 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683454 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9028.224521254 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1186767741389824 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157111533 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157111527 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1206522688524288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1015226919634944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1187569951813204 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683454 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9028.224521254 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1186767741389824 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_byteZÕþ s.all.current‚76627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1206632460558336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884465889272587e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198590181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _runtime‚9028.972341948 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1186875681095168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1042372606071296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19759406077440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19726659710572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27053298382848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175714223 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157125769 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157125763 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1167954086332416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175713759 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1167954060904448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1206635087172608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19756805854720 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1042372682685952 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27053346863104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101945645 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1015319335822848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19756805854720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175713759 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1167954060904448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1206635087172608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19756805854720 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1042372682685952 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27053346863104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101945645 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1015319335822848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19756805854720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1206632460558336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 % _timestamp‚1.7884465889272587e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198590181 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _runtime‚9028.972341948 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1186875681095168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1042372606071296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19759406077440 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19726659710572 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27053298382848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175714223 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157125769 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157125763 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1167954086332416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19849204584448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192164700265472 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192164726657024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74099049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1047023092505600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176494836 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19849204584448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27176742061056 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1173164803871744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1212011304640512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all §^TÎ.allocated‚1192981114964048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19846604375040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157823327 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19813712807636 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176495301 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102396249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357290266 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157823333 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310306 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 trainer/global_step‚10699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19846604375040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102396252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157823333 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884466276008463e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1019846350444544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199486973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚877 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157823327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74098587 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1019846322310144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1173164778443776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19816311092304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 _runtime‚9067.645754002 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1212013931241472 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1212011304640512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1047023017974784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199466939 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192164726657024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192164700265472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27176695664640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776[ü…?((ö'¢P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102396252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157823333 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884466276008463e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1019846350444544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199486973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚877 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157823327 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74098587 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1019846322310144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1173164778443776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19816311092304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 _runtime‚9067.645754002 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1212013931241472 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1212011304640512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1047023017974784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199466939 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192164726657024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192164700265472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27176695664640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19849204584448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192164700265472 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192164726657024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74099049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1047023092505600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176494836 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19849204584448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27176742061056 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1173164803871744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1212011304640512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1192981114964048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19846604375040 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157823327 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19813712807636 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176495301 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102396249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357290266 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157823333 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310306 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 trainer/global_step‚10699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19846604375040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074í¶¥›2÷'- _wandbruntime‚9068  _runtime‚9068î9 -'ø'§N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1192272639970816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199485173 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157837569 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192272666362368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884466283489738e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1212123703288832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74105571 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1019938738498048 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199505233 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19815542150304 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176510579 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1173271145046016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176511019 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157837569 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27179260180992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚9068.394188194 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1193086661768352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1212121076674560 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74105134 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157837563 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27179213797888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 trainer/global_step‚10699 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1212123703288832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157837563 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157837569 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192272666362368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884466283489738e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1212123703288832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74105571 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1019938738498048 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199505233 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19815542150304 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176510579 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1173271145046016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176511019 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157837569 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27179260180992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _runtime‚9068.394188194 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1193086661768352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1212121076674560 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74105134 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157837563 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27179213797888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 trainer/global_step‚10699 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1212123703288832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚74517504 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157837563 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199485173 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 % _timestamp‚1.7884466668316092e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200402025 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19905193532244 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 trainer/global_step‚10749 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200381991 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚483 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74434734 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1024465724985344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19940835433472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1178481837157376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1217502547357696 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158535127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177290303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74434254 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1217502547357696 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1217499920756736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1197561685532672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158535133 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19940835433472 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19938235224064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177290786 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158535133 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1197561711924224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 % _timestamp‚1.7884466668316092e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200402025 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19905193532244 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 trainer/global_step‚10749 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200381991 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚483 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74434734 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1024465724985344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19940835433472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1178481837157376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1217502547357696 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158535127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177290303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74434254 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1217502547357696 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1217499920756736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1197561685532672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158535133 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19940835433472 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19938235224064 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177290786 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158535133 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1197561711924224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102865248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚10749 _runtime‚9107.620649788 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19942667775488 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177306681 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200400225 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27304995828736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158549369 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1051863185482240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158549363 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024558141173248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200420285 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19940067552768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1197669651629568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19909621159788 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 _step‚882 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1217609692790784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177306219 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1197669625238016 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1178588203759616 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19942667775488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19940067552768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024558113038848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19907022874912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27305044308992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1197669625238016 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158549369 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1197669651629568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1198495201206560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158549363HÝÔ2'‹(¬N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1217612319405056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158549363 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024558141173248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200420285 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19940067552768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1197669651629568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19909621159788 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 _step‚882 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1217609692790784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177306219 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1197669625238016 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1178588203759616 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19942667775488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19940067552768 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024558113038848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19907022874912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27305044308992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1197669625238016 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158549369 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1197669651629568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1198495201206560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158549363 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102865248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 trainer/global_step‚10749 _runtime‚9107.620649788 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19942667775488 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177306681 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200400225 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27304995828736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158549369 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1051863185482240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1029085127660544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201297043 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360543970 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20032466282496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103315852 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1183798895870976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20029866073088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74769310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201297043 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103315849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1056512789449216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚10799 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1222991163473920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1222988536872960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1183798921298944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649OÜ0à((˜(¢P trainer/global_step‚10799 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1222991163473920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1222988536872960 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1183798921298944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1222991163473920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1203798193840464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159246927 _step‚885 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1202958697191424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20029866073088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1029085127660544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201297043 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360543970 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20032466282496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103315852 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1183798895870976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20029866073088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74769310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201297043 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103315849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1056512789449216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0õò2™(- _wandbruntime‚9146  _runtime‚9146E:Tß-'š(§N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178101425 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1056607756470272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19998503599520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1223100935521280 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1203066610505216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20031698401792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27430212621824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1203903740644768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201315277 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1183905262473216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20001101884396 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1056607679855616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201315277 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1223098308907008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20031698401792 trainer/global_step‚10799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159261163 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201335337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159261163 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74776377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74775927 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1029177515714048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103325048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201335337 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159261169 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1183905237045248 _step‚886 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20034298624512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159261169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1203066636896768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1203906364357612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1223100935521280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178100972 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1203066636896768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27430164141568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 _runtime‚9146.972690747 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20034298624512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1029177543848448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074á«ÄæÖ›(¬N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1223098308907008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20031698401792 trainer/global_step‚10799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159261163 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201335337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159261163 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74776377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74775927 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1029177515714048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103325048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201335337 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159261169 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1183905237045248 _step‚886 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20034298624512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159261169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1203066636896768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1203906364357612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1223100935521280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178100972 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1203066636896768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27430164141568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 _runtime‚9146.972690747 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20034298624512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1029177543848448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178101425 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1056607756470272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19998503599520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1223100935521280 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1203066610505216 LDDeviceStatsMon–Zµ\ itor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20031698401792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27430212621824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1203903740644768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201315277 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1183905262473216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20001101884396 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1056607679855616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201315277 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ö—°-2œ(- _wandbruntime‚9146  _runtime‚9146ólÓ^(Ø 0loss/FRt1/assignment_loss‚1.0760809694017683 2loss/SRqqt1/assignment_loss‚0.8567970617402807 2loss/SRqqt4/assignment_loss‚1.3081081621397364 1loss/SRqqt4/detection_loss‚1.9915449131901066 /loss/FBt1/detection_loss‚0.6485564107352817 %loss/total_loss‚38.65910253725369 .loss/FRt4/detection_loss‚2.183038400279151 trainer/global_step‚10799 /loss/FRt3/assignment_loss‚2.955715736438488 2loss/SRqqt2/assignment_loss‚1.4356880846326467 1loss/FBt1/assignment_loss‚0.09268212094668245 1loss/FBt2/assignment_loss‚0.08963682509117159 /loss/FRt1/detection_loss‚1.5807839554867573 /loss/FRt2/detection_loss‚2.2641156043570776 /loss/FBt2/detection_loss‚2.5795426964759827 _runtime‚9146.973841647 0loss/FRt2/assignment_loss‚1.9262056465332325 2loss/FBt3/assignment_loss‚0.061002249021839816 1loss/SRqqt1/detection_loss‚1.0190685468260199 epoch‚8 _step‚887 .loss/FBt4/detection_loss‚2.701642908155918 2loss/SRqqt3/assignment_loss‚0.9897782011674001 /loss/FRt3/detection_loss‚2.6142571915808164 1loss/SRqqt2/detection_loss‚1.5985406347718396 1loss/SRqqt3/detection_loss‚2.6949360141387353 .loss/FBt3/detection_loss‚2.868108093738556 $ _timestamp‚1.788446706929245e+09 /loss/FRt4/assignment_loss‚2.594089889526367 0loss/FBt4/assignment_loss‚0.5291822208778285ý²¹åcž(Ý 2loss/SRqqt1/assignment_loss‚0.8567970617402807 2loss/SRqqt4/assignment_loss‚1.3081081621397364 1loss/SRqqt4/detection_loss‚1.9915449131901066 /loss/FBt1/detection_loss‚0.6485564107352817 %loss/total_loss‚38.65910253725369 .loss/FRt4/detection_loss‚2.183038400279151 trainer/global_step‚10799 /loss/FRt3/assignment_loss‚2.955715736438488 2loss/SRqqt2/assignment_loss‚1.4356880846326467 1loss/FBt1/assignment_loss‚0.09268212094668245 1loss/FBt2/assignment_loss‚0.08963682509117159 /loss/FRt1/detection_loss‚1.5807839554867573 /loss/FRt2/detection_loss‚2.2641156043570776 /loss/FBt2/detection_loss‚2.5795426964759827 _runtime‚9146.973841647 0loss/FRt2/assignment_loss‚1.9262056465332325 2loss/FBt3/assignment_loss‚0.061002249021839816 1loss/SRqqt1/detection_loss‚1.0190685468260199 epoch‚8 _step‚887 .loss/FBt4/detection_loss‚2.701642908155918 2loss/SRqqt3/assignment_loss‚0.9897782011674001 /loss/FRt3/detection_loss‚2.6142571915808164 1loss/SRqqt2/detection_loss‚1.5985406347718396 1loss/SRqqt3/detection_loss‚2.6949360141387353 .loss/FBt3/detection_loss‚2.868108093738556 $ _timestamp‚1.788446706929245e+09 /loss/FRt4/assignment_loss‚2.594089889526367 0loss/FBt4/assignment_loss‚0.5291822208778285 0loss/FRt1/assignment_loss‚1.0760809694017683÷TÆ韟(‚(:” õæÔóåÉ gpu.0.gpu‚94.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.598' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.519600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;£ÆL8 (‚(:­ õæÔÓŲË disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1772.9609375 disk.dm-1.out‚ 335.546875 disk.sda3.in‚0.5625 disk.sda.out‚ 335.546875 disk.sda2.in‚0% disk.nvme0n1.out‚75669.2978515625# memory_percent‚88.5857105255127 network.recv‚ 3.320794e+06' disk.nvme0c0n1.out‚75669.2978515625 disk.dm-0.in‚ 1772.9609375" disk.dm-0.out‚75669.2978515625 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2725.61328125 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625( proc.memory.availableMB‚ 1870.1171875 proc.cpu.threads‚17) disk./.usagePercent‚74.38117833222684$ disk./.usageGB‚1108.333309173584 disk.rbd0.in‚0 disk.nvme0n1.in‚ 1772.9609375 disk.sda3.out‚ 335.546875 disk.rbd2.out‚645.7939453125 disk.sda1.in‚0 cpu‚25.312078737245162 network.sent‚ 2.7136247e+07 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.662109375 disk.rbd3.out‚0* proc.memory.percent‚16.635823249816895íYuGž¡(‚(:“ „æÔ…±…É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚281.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òyw»A¢(‚(:¶ „æÔº“ÆË disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 1773.6796875 network.recv‚ 3.327513e+06 disk.sda1.in‚0# disk.dm-0.out‚75670.09716796875$ memory_percent‚88.58575820922852* proc.memory.percent‚16.635823249816895 network.sent‚ 2.724512e+07 disk.rbd2.out‚647.3837890625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1773.6796875 cpu‚25.311677366454326 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.nvme0n1.in‚ 1773.6796875' proc.memory.availableMB‚ 1870.109375# proc.memory.rssMB‚ 2725.61328125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚75670.09716796875( disk.nvme0c0n1.out‚75670.09716796875 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 4.662109375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 336.04296875 proc.cpu.threads‚17) disk./.usagePercent‚74.38119036458095% disk./.usageGB‚1108.3334884643555 disk.sda.out‚ 336.04296875 disk.sda3.in‚0.5625 disk.sda3.out‚ 336.04296875 disk.rbd0.in‚0ÎzX”£(‚(:‰ “æÔºä¢É gpu.0.gpu‚68.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.876' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.9752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞØi=¤(‚(:² “æÔì«¥Ì disk.sda3.in‚0.5625 disk.rbd0.out‚0) proc.memory.availableMB‚ 1869.87109375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚647.4462890625 disk.sda1.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚75673.9892578125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 1773.6796875" disk.nvme0c0n1.in‚ 1773.6796875 disk.dm-1.in‚0.5625% disk./.usageGB‚1108.3366508483887 disk.rbd1.in‚0$ memory_percent‚88.58721256256104) disk./.usagePercent‚74.38140259482682 disk.sda.out‚ 336.7421875 disk.sda2.out‚0' disk.nvme0c0n1.out‚75673.9892578125 disk.dm-0.in‚ 1773.6796875" disk.dm-0.out‚75673.9892578125# proc.memory.rssMB‚ 2725.61328125* proc.memory.percent‚16.635823249816895 disk.rbd3.out‚0 disk.sda3.out‚ 336.7421875 disk.dm-1.out‚ 336.7421875 cpu‚25.311246856276234 network.recv‚ 3.33234e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 network.sent‚ 2.7253527e+07 disk.rbd1.out‚ 4.662109375 disk.sda.in‚0.5625÷é Ü2¥(- _wandbruntime‚9185  _runtime‚9185äI÷mʦ(Ä ' lr-AdamW/pg2‚1.2305008043416439e-05 trainer/global_step‚10849 % _timestamp‚1.7884467455335097e+09 _runtime‚9185.578211104 _step‚888 ' lr-AdamW/pg1‚1.2305008043416439e-05manϧ(É ' lr-AdamW/pg2‚1.2305008043416439e-05 trainer/global_step‚10849 % _timestamp‚1.7884467455335097e+09 _runtime‚9185.578211104 _step‚888 ' lr-AdamW/pg1‚1.2305008043416439e-05øuì¢2¨(- _wandbruntime‚9185  _runtime‚9185Ú¼2–#(©(P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚474 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884467455344415e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103775649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1228477152989184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20124097131520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20121496922112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9185.579302111 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1228479779590144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1061257620850688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1228477152989184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1208355656067072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1033704558470144 _step‚889 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362170822 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159958727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202212095 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1209204109566036 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1189115980012544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178879600 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27553062380544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1208355682458624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362190862 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75103951 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209206733278672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20090753266128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159958733 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20124097131520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159958733 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20088154981460 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178880077 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776¿/:((ª(¢P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159958733 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20124097131520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159958733 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20088154981460 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178880077 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚474 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884467455344415e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103775649 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1228477152989184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20124097131520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20121496922112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9185.579302111 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1228479779590144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1061257620850688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1228477152989184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1208355656067072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1033704558470144 _step‚889 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362170822 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159958727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202212095 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1209204109566036 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1189115980012544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178879600 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27553062380544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1208355682458624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362190862 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75103951 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209206733278672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20090753266128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400ùqvÕÞ2«(- _wandbruntime‚9186  _runtime‚9186ì=ú2¬(§N F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚890 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20125929473536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202250389 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1208463622163968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178895902 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209312280082976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1061352443924480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1189222321186816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159972963 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1228589551637504 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.fÊÁýûreed‚ 159972963 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223358 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178895439 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10849 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1228589551637504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20123329250816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75111054 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1061352520539136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202230329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1189222295758848 _runtime‚9186.334368694 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103784845 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159972969 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159972969 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20089984324128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209314903795820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20123329250816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1208463595772416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884467462893138e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202230329 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223358 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1033796946523648 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27555574015488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20092582609004 îÆ~2'­(¬N FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209314903795820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20123329250816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1208463595772416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884467462893138e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202230329 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223358 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1033796946523648 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27555574015488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20092582609004 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 _step‚890 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20125929473536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202250389 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1208463622163968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178895902 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209312280082976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1061352443924480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1189222321186816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159972963 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1228589551637504 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159972963 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223358 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178895439 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10849 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1228589551637504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20123329250816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75111054 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1061352520539136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202230329 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1189222295758848 _runtime‚9186.334368694 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103784845 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159972969 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159972969 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20089984324128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074úíï~2®(- _wandbruntime‚9186  _runtime‚9186A q^¯(Ø 3loss/FBt3/assignment_loss‚0.0011403518101360532 0loss/FBt1/detection_loss‚0.44185051845974826 2loss/SRqqt1/assignment_loss‚1.0924385153437015 2loss/SRqqt3/assignment_loss‚1.1513288114697506 0loss/FBt2/assignment_loss‚0.5353189291839954 .loss/FRt4/detection_loss‚2.627393977273078 /loss/FRt2/assignment_loss‚2.233225429759306 1loss/SRqqt1/detection_loss‚0.9436351305339485 1loss/SRqqt3/detection_loss‚2.3095278159165993 _runtime‚9186.335868732 /loss/FRt1/detection_loss‚1.5077711568867906 .loss/FRt3/detection_loss‚2.393426094814486 trainer/global_step‚10849 1loss/SRqqt2/assignment_loss‚1.172080005756488 .loss/FRt2/detection_loss‚2.471837798023925 /loss/FBt3/detection_loss‚3.7178482711315155 epoch‚8 /loss/FRt3/assignment_loss‚2.994014440160809 1loss/SRqqt2/detection_loss‚1.5487855044650096 1loss/SRqqt4/detection_loss‚1.8714015077460895 /loss/FBt2/detection_loss‚2.3251171946525573 .loss/FBt4/detection_loss‚2.376585841178894 $ _timestamp‚1.788446746290934e+09 _step‚891 0loss/FRt1/assignment_loss‚1.2468150575110253 1loss/FBt1/assignment_loss‚0.07803605258291997 2loss/FBt4/assignment_loss‚0.014686648438717904 &loss/total_loss‚38.690845543896955 /loss/FRt4/assignment_loss‚2.536616257258824 1loss/SRqqt4/assignment_loss‚1.099964233538644«ßxc°(Ý 2loss/SRqqt3/assignment_loss‚1.1513288114697506 0loss/FBt2/assignment_loss‚0.5353189291839954 .loss/FRt4/detection_loss‚2.627393977273078 /loss/FRt2/assignment_loss‚2.233225429759306 1loss/SRqqt1/detection_loss‚0.9436351305339485 1loss/SRqqt3/detection_loss‚2.3095278159165993 _runtime‚9186.335868732 /loss/FRt1/detection_loss‚1.5077711568867906 .loss/FRt3/detection_loss‚2.393426094814486 trainer/global_step‚10849 1loss/SRqqt2/assignment_loss‚1.172080005756488 .loss/FRt2/detection_loss‚2.471837798023925 /loss/FBt3/detection_loss‚3.7178482711315155 epoch‚8 /loss/FRt3/assignment_loss‚2.994014440160809 1loss/SRqqt2/detection_loss‚1.5487855044650096 1loss/SRqqt4/detection_loss‚1.8714015077460895 /loss/FBt2/detection_loss‚2.3251171946525573 .loss/FBt4/detection_loss‚2.376585841178894 $ _timestamp‚1.788446746290934e+09 _step‚891 0loss/FRt1/assignment_loss‚1.2468150575110253 1loss/FBt1/assignment_loss‚0.07803605258291997 2loss/FBt4/assignment_loss‚0.014686648438717904 &loss/total_loss‚38.690845543896955 /loss/FRt4/assignment_loss‚2.536616257258824 1loss/SRqqt4/assignment_loss‚1.099964233538644 3loss/FBt3/assignment_loss‚0.0011403518101360532 0loss/FBt1/detection_loss‚0.44185051845974826 2loss/SRqqt1/assignment_loss‚1.0924385153437015û’Û°”±(‚(:‰ ¢æÔɧóÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€É}÷C²(‚(:¸ ¢æÔ– “Ì disk.sda3.out‚ 337.37109375 disk.dm-0.in‚ 1773.6796875) proc.memory.availableMB‚ 1869.83984375 cpu‚25.31082267065892 disk.sda.in‚0.5625 disk.rbd1.out‚ 4.662109375 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.rbd2.out‚647.4580078125 disk.rbd3.out‚0# disk.dm-0.out‚75936.23095703125# proc.memory.rssMB‚ 2725.61328125 network.sent‚ 2.7256682e+07 network.recv‚ 3.336125e+06 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1773.6796875) disk./.usagePercent‚74.31185968419565 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 337.37109375* proc.memory.percent‚16.635823249816895 disk.rbd2.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚75936.23095703125( disk.nvme0c0n1.out‚75936.23095703125 disk.dm-1.out‚ 337.37109375 disk.drbd1777.out‚0$ memory_percent‚88.58740329742432 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0% disk./.usageGB‚1107.3004112243652 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1773.6796875¹#Ï¿Ÿ³(‚(:” ±æÔ¬˜éÈ gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚259.259' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.851800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U:‰A´(‚(:¶ ±æÔ¢¾âË disk.rbd2.in‚0 cpu‚25.310374293559445 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 338.32421875 disk.sda1.out‚0* proc.memory.percent‚16.635823249816895 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.rbd1.out‚ 4.662109375 disk.sda.out‚ 338.32421875 network.sent‚ 2.7364754e+07# disk.nvme0c0n1.in‚ 1776.61328125 disk.dm-1.in‚0.5625 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1776.61328125$ memory_percent‚88.58747482299805) disk./.usagePercent‚74.31186531636142' proc.memory.availableMB‚ 1869.828125 network.recv‚ 3.34184e+06' disk.nvme0c0n1.out‚75939.4189453125 disk.dm-0.in‚ 1776.61328125" disk.dm-0.out‚75939.4189453125! disk.rbd2.out‚647.54443359375 disk.sda2.in‚0% disk.nvme0n1.out‚75939.4189453125 disk.sda3.out‚ 338.32421875 disk.sda1.in‚0# proc.memory.rssMB‚ 2725.61328125$ disk./.usageGB‚1107.300495147705 disk.rbd0.in‚04%ûU“µ(‚(:ˆ ÀæÔŽ¿¸É gpu.0.gpu‚91.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚39.0 gpu.0.powerWatts‚90.248' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ýôø?¶(‚(:´ ÀæÔÿ©åË disk.rbd3.out‚0( disk./.usagePercent‚74.3118420196758 disk.dm-0.in‚ 1776.62109375 disk.rbd0.in‚0 cpu‚25.317490930025414 network.recv‚ 3.346041e+06" disk.dm-0.out‚75941.6298828125 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚75941.6298828125 disk.sda3.out‚ 338.7890625 disk.rbd1.in‚0$ memory_percent‚88.58485221862793( proc.memory.availableMB‚ 1870.2578125# proc.memory.rssMB‚ 2725.61328125 disk.dm-1.out‚ 338.7890625 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ disk./.usageGB‚1107.300148010254 disk.rbd1.out‚ 4.662109375! disk.rbd2.out‚647.56005859375 disk.sda.out‚ 338.7890625# disk.nvme0c0n1.in‚ 1776.62109375' disk.nvme0c0n1.out‚75941.6298828125 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1776.62109375 disk.sda3.in‚0.5625 network.sent‚ 2.7370583e+07 disk.drbd1777.in‚0* proc.memory.percent‚16.635823249816895 proc.cpu.threads‚17õh}I“·(‚(:ˆ ÏæÔÞ›É gpu.0.gpu‚91.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËR9C¸(‚(:¸ ÏæÔíûËÎ proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.drbd1777.out‚0! disk.rbd2.out‚649.19677734375$ memory_percent‚88.58494758605957 network.sent‚ 2.7376828e+07 network.recv‚ 3.350285e+06 disk.sda.in‚0.5625 disk.sda.out‚ 339.4296875 disk.rbd3.in‚0 disk.sda3.out‚ 339.4296875( disk.nvme0c0n1.out‚75942.72607421875# proc.memory.rssMB‚ 2725.61328125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 disk.rbd2.in‚0 cpu‚25.32486549988955 disk.sda2.out‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.dm-0.in‚ 1777.49609375 disk.dm-1.out‚ 339.4296875 disk.sda1.out‚0& disk.nvme0n1.out‚75942.72607421875* proc.memory.percent‚16.635823249816895 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1777.49609375) disk./.usagePercent‚74.31183408344224% disk./.usageGB‚1107.3000297546387 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1777.49609375# disk.dm-0.out‚75942.72607421875( proc.memory.availableMB‚ 1870.2421875lmk5¹(‚(:… ÞæÔ¢àýÈ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚36.0 gpu.0.powerWatts‚88.545' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.709 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ÚͼBº(‚(:· ÞæÔÊØ»Ê disk.sda2.out‚0% disk.nvme0n1.out‚75943.6416015625 disk.sda3.in‚0.5625 network.sent‚ 2.738208e+07' disk.nvme0c0n1.out‚75943.6416015625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2725.61328125" disk.dm-0.out‚75943.6416015625) disk./.usagePercent‚74.31185610009018 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1777.49609375* proc.memory.percent‚16.635823249816895 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 disk.rbd2.in‚0 disk.dm-0.in‚ 1777.49609375 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 339.97265625 network.recv‚ 3.353916e+06! disk.rbd2.out‚649.21240234375 disk.sda.out‚ 339.97265625 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1777.49609375$ memory_percent‚88.58537673950195% disk./.usageGB‚1107.3003578186035 disk.sda1.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 1870.171875 cpu‚25.331959474994104 disk.drbd1777.out‚0 disk.sda3.out‚ 339.97265625 disk.rbd0.out‚0 disk.sda.in‚0.5625k·˜y2»(- _wandbruntime‚9262  _runtime‚9262$À¼(†U aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚17648640 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 23672651776 % _timestamp‚1.7884468225326855e+09 fQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚1215423923836928 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚141 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 23672651776 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 bMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚1215423899542528 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚20205988553216 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚75543144 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 cNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚1215423923836928 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975012864 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2649 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 364117935 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1195948928197632 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 160948086 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚1216121427358956 [FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚1216119455225492 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1235627937377280 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1039698093900800 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 364101956 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1215423899542528 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23672651776 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1575 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 cNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚1215423923836928 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 1975012864 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚2649 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 364117935 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1438 bMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚1195948928197632 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 160948086 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚1216121427358956 [FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚1216119455225492 XCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚1235627937377280 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1039698093900800 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 1948802648 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24124 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 364101956 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 _JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚1215423899542528 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 23672651776 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1575 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1235702170168320 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 104466424 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2649 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1575 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚27657637455872 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚20170999713748 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23672651776 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 364120064 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚27657520382464 `KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚1196018929298432 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 203161084 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 364136043 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 dODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚1196018952629248 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1067417596637696 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 203177058 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚20206462756352 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 1975012864 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 180014220 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 104466422 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚141 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚20204512037888 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 364120064 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚1235702170168320 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 104466424 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2649 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1575 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚27657637455872 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚20170999713748 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 23672651776 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 364120064 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚27657520382464 `KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚1196018929298432 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 203161084 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 364136043 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 1950718464 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 dODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚1196018952629248 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1067417596637696 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 203177058 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚20206462756352 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 1975012864 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 180014220 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 104466422 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚141 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2ýv/³ã‘Á(‚(:† íæÔ×°®É gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚37.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È¥_ÜAÂ(‚(:¶ íæÔ˜’îË disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 340.6484375 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚75944.1962890625( proc.memory.availableMB‚ 1870.9765625* proc.memory.percent‚16.635823249816895 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 340.6484375 disk.sda3.out‚ 340.6484375 cpu‚25.338730258846542! disk.rbd2.out‚649.33740234375 disk.rbd3.out‚0 disk.rbd0.in‚0$ memory_percent‚88.58046531677246 network.recv‚ 3.358308e+06) disk./.usagePercent‚74.31186122024086 disk.rbd1.out‚ 4.662109375 disk.sda.in‚0.5625 disk.sda3.in‚0.5625 disk.rbd3.in‚0' disk.nvme0c0n1.out‚75944.1962890625# proc.memory.rssMB‚ 2725.61328125 network.sent‚ 2.7388074e+07 disk.dm-0.in‚ 1778.21484375 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.21484375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1778.21484375% disk./.usageGB‚1107.3004341125488" disk.dm-0.out‚75944.1962890625ËUƒx2Ã(- _wandbruntime‚9277  _runtime‚9277Š3->aÄ(¶§ _step‚894 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874865412712097 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03742138296365738 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5352182984352112 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8498986959457397 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1610034555196762 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1465943157672882 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5386262536048889 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4233333170413971 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4169963300228119 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49545818567276 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912883639335632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3888888359069824 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5952600836753845 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5601304769515991 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9845110177993774 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.022197028622031212 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13965517282485962 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.31111112236976624 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.46045759320259094 jet/accuracy_4_of_7‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.615998387336731 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14997315406799316 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2692524492740631 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5315038561820984 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9918732643127441 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012858856469392776 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15037429332733154 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2693922519683838 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21370603144168854 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17596733570098877 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_3_of_7‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5192197561264038 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2807229459285736 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01915040798485279 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37285757064819336 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 *loss/val_total_loss‚40.091789460726865 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03015304170548916 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07960813492536545 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 trainer/global_step‚10880 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19047211110591888 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.045275893062353134 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7291666269302368 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1269005835056305 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.11934524029493332 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.017153097316622734 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015211639925837517 9"Purity/1FRt2SRqqt0FBt/event_purity‚0.0023547881282866 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5285786390304565 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899652004241943 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21567347645759583 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3618553876876831 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.142435222864151 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8179594278335571 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20814324915409088 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5341880917549133 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5440234541893005 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886142611503601 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9474369287490845 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 _runtime‚9277.215146578 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5283617377281189 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 particle/accuracy_3_of_6‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7057093381881714 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5590278506278992 jet/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10305989533662796 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.01773049682378769 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 epoch‚8 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5141398906707764 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16077396273612976 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4270972013473511 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5629439949989319 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5200023055076599 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11658678203821182 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35401931405067444 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9901229739189148 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2563425302505493 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17621156573295593 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11500188708305359 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5704003572463989 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12564712762832642 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20157426595687866 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900158047676086 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.13425880670547485 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1257065385580063 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.49429425597190857 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5490527153015137 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27201223373413086 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13046105206012726 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11910142004489899 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12848159670829773 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.173137828707695 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10759636759757996 -jet/accuracy_4_of_4‚0.0028737422544509172 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17689144611358643 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35776761174201965 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8333333730697632 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0.5 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994656503200531 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5150002837181091 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35357144474983215 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4837847054004669 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5068917274475098 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08014360815286636 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1717669665813446 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5840359926223755 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5652152895927429 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4780701696872711 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5053872466087341 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.030683567747473717 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5763888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14532075822353363 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 *jet/accuracy_1_of_5‚0.9597285985946655 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2712860107421875 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46522989869117737 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25326985120773315 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7validation_average_jet_accuracy‚0.44222700595855713 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.554515540599823 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4382716417312622 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31704944372177124 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991768658161163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_3_of_4‚0.0729934573173523 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9976640343666077 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3465277850627899 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 % _timestamp‚1.7884468371684163e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2785611152648926 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4940612316131592 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5213474631309509 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.529539942741394 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.51845782995224 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5649304986000061 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07043761014938354 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17621156573295593 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17222250998020172 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22615811228752136 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176245033740997 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.512740433216095 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5100193619728088 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 *jet/accuracy_2_of_4‚0.3868028521537781 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.48552513122558594 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909427762031555 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19256757199764252 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4526144564151764 particle/accuracy_1_of_7‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 jet/accuracy_5_of_7‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.21057768166065216 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.01984127052128315 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993831992149353 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4208482503890991 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5260518789291382 *jet/accuracy_1_of_3‚0.8757859468460083 ,jet/accuracy_3_of_3‚0.029662953689694405 +jet/accuracy_3_of_5‚0.14062340557575226 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5728257298469543 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22376543283462524 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5600000023841858 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_p„ƒ2roportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14000001549720764 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06752873957157135 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.41375967860221863 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972668886184692 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5453009605407715 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5682993531227112 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17929291725158691 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5403879880905151 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48993638157844543 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2109384685754776 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03390643000602722 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7564452886581421 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4097222089767456 particle/accuracy_2_of_6‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45187777280807495 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2583453357219696 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5493589639663696 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.31382685899734497 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 jet/accuracy_5_of_5‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5221901535987854 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823284149169922 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5487167835235596 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5262802839279175 particle/accuracy_4_of_5‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49993640184402466 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890661239624023 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5771185755729675 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04592181369662285 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_1_of_1‚0.49053770303726196 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961973428726196 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4920077323913574 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 +jet/accuracy_3_of_6‚0.11538461595773697 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6453564763069153 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10928609222173691 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 +particle/sensitivity‚0.3229414224624634 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5675052404403687 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1404956430196762 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4097222089767456 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632002353668 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.002116402145475149 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_4‚0.9287684559822083 'particle/f_score‚0.4017430543899536 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1887674182653427 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 particle/accuracy_2_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.51845782995224 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015318864956498146 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008897725492715836 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3181818127632141 particle/accuracy_3_of_3‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 jet/accuracy_1_of_7‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4251174032688141 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4122768044471741 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5257055163383484 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5812498927116394 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6517857313156128 +jet/accuracy_2_of_2‚0.11437152326107025 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39487770199775696 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5059283971786499 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9977324604988098 particle/accuracy_3_of_4‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.040365103632211685 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9926454424858093 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6094248294830322 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21339064836502075 particle/accuracy_1_of_6‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9867132306098938 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4238394796848297 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3005976676940918 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 +jet/accuracy_2_of_6‚0.49358975887298584 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933807253837585 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7291666269302368 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2882183790206909 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5301339626312256 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3478836119174957 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9816809892654419 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21513731777668 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1aÛùsMÅ(»§ 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07043761014938354 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17621156573295593 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17222250998020172 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22615811228752136 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176245033740997 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.512740433216095 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5100193619728088 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 *jet/accuracy_2_of_4‚0.3868028521537781 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.48552513122558594 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909427762031555 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19256757199764252 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4526144564151764 particle/accuracy_1_of_7‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 jet/accuracy_5_of_7‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.21057768166065216 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt0SRqqt3FBt/event_purity‚0.01984127052128315 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993831992149353 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4208482503890991 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5260518789291382 *jet/accuracy_1_of_3‚0.8757859468460083 ,jet/accuracy_3_of_3‚0.029662953689694405 +jet/accuracy_3_of_5‚0.14062340557575226 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5728257298469543 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22376543283462524 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5600000023841858 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14000001549720764 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.06752873957157135 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.41375967860221863 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972668886184692 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5453009605407715 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5682993531227112 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17929291725158691 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5403879880905151 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48993638157844543 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2109384685754776 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03390643000602722 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7564452886581421 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4097222089767456 particle/accuracy_2_of_6‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45187777280807495 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2583453357219696 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5493589639663696 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.31382685899734497 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 jet/accuracy_5_of_5‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5221901535987854 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823284149169922 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5487167835235596 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5262802839279175 particle/accuracy_4_of_5‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49993640184402466 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4629629850387573 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890661239624023 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5771185755729675 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04592181369662285 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_1_of_1‚0.49053770303726196 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8030303716659546 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961973428726196 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4920077323913574 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 +jet/accuracy_3_of_6‚0.11538461595773697 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6453564763069153 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10928609222173691 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 +particle/sensitivity‚0.3229414224624634 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5675052404403687 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1404956430196762 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4097222089767456 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632002353668 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.002116402145475149 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_4‚0.9287684559822083 'particle/f_score‚0.4017430543899536 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1887674182653427 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 particle/accuracy_2_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.51845782995224 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015318864956498146 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008897725492715836 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3181818127632141 particle/accuracy_3_of_3‚1 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 jet/accuracy_1_of_7‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4251174032688141 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4122768044471741 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5257055163383484 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5812498927116394 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6517857313156128 +jet/accuracy_2_of_2‚0.11437152326107025 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39487770199775696 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5059283971786499 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9977324604988098 particle/accuracy_3_of_4‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.040365103632211685 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9926454424858093 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6094248294830322 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21339064836502075 particle/accuracy_1_of_6‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9867132306098938 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4238394796848297 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3005976676940918 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 +jet/accuracy_2_of_6‚0.49358975887298584 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 particle/accuracy_6_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933807253837585 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7291666269302368 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2882183790206909 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5301339626312256 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3478836119174957 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9816809892654419 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21513731777668 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 _step‚894 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874865412712097 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03742138296365738 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5352182984352112 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8498986959457397 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1610034555196762 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1465943157672882 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5386262536048889 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4233333170413971 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4169963300228119 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49545818567276 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912883639335632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3888888359069824 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5952600836753845 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5601304769515991 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9845110177993774 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_ŒjOFproportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.022197028622031212 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13965517282485962 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.31111112236976624 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.46045759320259094 jet/accuracy_4_of_7‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.615998387336731 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14997315406799316 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2692524492740631 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5315038561820984 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9918732643127441 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012858856469392776 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15037429332733154 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2693922519683838 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21370603144168854 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17596733570098877 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 jet/accuracy_3_of_7‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5192197561264038 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2807229459285736 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01915040798485279 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37285757064819336 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 *loss/val_total_loss‚40.091789460726865 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03015304170548916 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07960813492536545 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 trainer/global_step‚10880 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19047211110591888 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.045275893062353134 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7291666269302368 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1269005835056305 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.11934524029493332 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.017153097316622734 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909092128276825 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015211639925837517 9"Purity/1FRt2SRqqt0FBt/event_purity‚0.0023547881282866 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5285786390304565 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899652004241943 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21567347645759583 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3618553876876831 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.142435222864151 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8179594278335571 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20814324915409088 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5341880917549133 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5440234541893005 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886142611503601 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9474369287490845 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 _runtime‚9277.215146578 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5283617377281189 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 particle/accuracy_3_of_6‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7057093381881714 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5590278506278992 jet/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10305989533662796 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.01773049682378769 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 epoch‚8 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5141398906707764 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16077396273612976 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4270972013473511 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5629439949989319 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5200023055076599 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11658678203821182 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35401931405067444 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9901229739189148 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2563425302505493 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17621156573295593 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11500188708305359 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5704003572463989 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12564712762832642 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20157426595687866 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900158047676086 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.13425880670547485 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1257065385580063 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.49429425597190857 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.5490527153015137 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27201223373413086 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13046105206012726 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11910142004489899 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12848159670829773 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.173137828707695 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10759636759757996 -jet/accuracy_4_of_4‚0.0028737422544509172 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17689144611358643 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35776761174201965 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8333333730697632 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0.5 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994656503200531 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5150002837181091 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35357144474983215 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4837847054004669 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5068917274475098 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08014360815286636 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1717669665813446 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5840359926223755 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5652152895927429 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4780701696872711 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5053872466087341 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.030683567747473717 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5763888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14532075822353363 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 *jet/accuracy_1_of_5‚0.9597285985946655 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2712860107421875 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46522989869117737 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25326985120773315 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7validation_average_jet_accuracy‚0.44222700595855713 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.554515540599823 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4382716417312622 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31704944372177124 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991768658161163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_3_of_4‚0.0729934573173523 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9976640343666077 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3465277850627899 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 % _timestamp‚1.7884468371684163e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2785611152648926 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4940612316131592 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5213474631309509 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.529539942741394 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.51845782995224 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5649304986000061þ;,wœÆ(‚(:‘ üæÔÔ«¬É gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚31.0 gpu.0.powerWatts‚75.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.119200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ÐûBÇ(‚(:· üæÔñâ«Ë proc.memory.rssMB‚ 3751.03125) proc.memory.percent‚22.89447784423828 disk.dm-1.in‚0.5625 proc.cpu.threads‚17 disk.sda.in‚0.5625 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.21484375& disk.nvme0n1.out‚77133.23876953125 network.sent‚ 2.7496868e+07 disk.rbd1.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 2680.296875! disk.rbd2.out‚649.35302734375 disk.rbd0.in‚0% disk./.usageGB‚1107.2991676330566 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.662109375 disk.sda.out‚ 341.02734375 disk.rbd3.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.31177622573951 disk.rbd2.in‚0 disk.dm-0.in‚ 1778.21484375# disk.dm-0.out‚77133.23876953125 cpu‚25.341597147227855 disk.drbd1777.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1778.21484375 disk.dm-1.out‚ 341.02734375$ memory_percent‚83.64076614379883 network.recv‚ 3.365174e+06 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 341.02734375( disk.nvme0c0n1.out‚77133.23876953125äÆ,“È(‚(:ˆ ‹‘æÔ÷¹êÈ gpu.0.gpu‚81.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó&Ãæ@É(‚(:µ ‹‘æÔ¶Ñ×Ë disk.rbd1.out‚ 4.662109375& disk.nvme0n1.out‚81460.46923828125 disk.sda3.in‚0.5625) disk./.usagePercent‚74.32348140222003 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚81460.46923828125* proc.memory.percent‚16.390228271484375 disk.sda2.out‚0 disk.sda3.out‚ 341.51953125 disk.sda.out‚ 341.51953125 network.recv‚ 3.371829e+06 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1778.21484375 network.sent‚ 2.7621729e+07% disk./.usageGB‚1107.4735832214355 disk.rbd0.in‚0# disk.dm-0.out‚81460.46923828125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1778.21484375( proc.memory.availableMB‚ 3746.7421875 cpu‚25.341072683112653 disk.dm-1.out‚ 341.51953125 disk.rbd1.in‚0$ memory_percent‚77.13170051574707 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 649.501953125 disk.dm-0.in‚ 1778.21484375 proc.memory.rssMB‚2685.375 proc.cpu.threads‚17>-=2Ê(- _wandbruntime‚9302  _runtime‚9302ŒÈMÉË(à _runtime‚9302.221052999 _step‚895 ' lr-AdamW/pg1‚1.1343040343950706e-05 ' lr-AdamW/pg2‚1.1343040343950706e-05 trainer/global_step‚10899 $ _timestamp‚1.788446862176387e+09ç$ÕÊÎÌ(È trainer/global_step‚10899 $ _timestamp‚1.788446862176387e+09 _runtime‚9302.221052999 _step‚895 ' lr-AdamW/pg1‚1.1343040343950706e-05 ' lr-AdamW/pg2‚1.1343040343950706e-05ÿÅìú'2Í(- _wandbruntime‚9302  _runtime‚9302æÌ "(Î(œP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20212030309058 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1238645074430464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1042228836898816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1198843082648576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1219052514672966 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1218399454661632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _step‚896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20243019559424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104735035 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104735032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1238642447829504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 _runtime‚9302.222281675 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203583566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75727578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75727128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161356855 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203603600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203583566 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27709609309184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180462613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1219055138385602 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180462160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1238642447829504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 _runtime‚9302.222281675 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203583566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75727578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75727128 GADeviceStatsMonitor.on_train_batch_start/oversizeØñkÙ‘$_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161356855 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203603600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203583566 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27709609309184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180462613 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1219055138385602 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180462160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1042228836898816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1198843082648576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1219052514672966 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1218399454661632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _step‚896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20243019559424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104735035 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104735032€:Ä Á2Ð(- _wandbruntime‚9302  _runtime‚9302´ÙZ,'Ñ(¦N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104744228 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20247452110848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161371091 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚413 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1219160685189906 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27712202555392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161371085 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27712160366592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180478552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180478139 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75734321 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1198949449250816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1218507367975424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992951 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203621860 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161371085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1219163308902750 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20244851888128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161371091 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1070033385318912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1238754846477824 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104744231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚410 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1218507367975424 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1042321253086720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20213859651934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75733911 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20211261367058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238754846477824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203601800 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1042321224952320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 trainer/global_step‚10899 % _timestamp‚1.7884468629169505e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203621860 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203601800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20244851888128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚ 9302.96208881 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 _step‚897 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364972885 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1218507394366976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1218507394366976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1198949423822848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575‚•1'Ò(«N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚42188800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1218507367975424 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1042321253086720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20213859651934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75733911 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20211261367058 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238754846477824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203601800 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1042321224952320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 trainer/global_step‚10899 % _timestamp‚1.7884468629169505e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203621860 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203601800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20244851888128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚ 9302.96208881 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 _step‚897 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364972885 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1218507394366976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1218507394366976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1198949423822848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104744228 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20247452110848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161371091 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚413 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚70323200 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1219160685189906 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27712202555392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161371085 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27712160366592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180478552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180478139 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75734321 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1198949449250816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1218507367975424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364992951 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203621860 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161371085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1219163308902750 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20244851888128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161371091 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1070033385318912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1238754846477824 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104744231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚410N³‹2Ó(- _wandbruntime‚9302  _runtime‚9302Ò2­Ð`Ô(Ú 1loss/FBt1/assignment_loss‚0.09856781850032974 /loss/FRt2/detection_loss‚2.1882120087742805 /loss/FRt3/detection_loss‚2.3541202602941276 _runtime‚9302.963727378 0loss/FRt1/assignment_loss‚1.2338943992342268 2loss/SRqqt2/assignment_loss‚1.3387822376719365 2loss/SRqqt3/assignment_loss‚0.9089231673904039 1loss/FBt2/assignment_loss‚0.24869223280984443 /loss/FRt4/detection_loss‚2.0891829046607016 0loss/SRqqt1/detection_loss‚0.904642808756658 2loss/SRqqt4/assignment_loss‚1.1943872924561478 1loss/FBt4/assignment_loss‚0.28494009931716846 1loss/SRqqt4/detection_loss‚1.6558421126433782 trainer/global_step‚10899 2loss/SRqqt1/assignment_loss‚0.6149466666193413 1loss/FBt3/assignment_loss‚0.05144746447331272 /loss/FBt3/detection_loss‚2.6165456250309944 1loss/SRqqt2/detection_loss‚1.6188871315370004 /loss/FRt2/assignment_loss‚1.790330994129181 /loss/FRt1/detection_loss‚1.8216173024404616 .loss/FBt2/detection_loss‚2.264351119597753 .loss/FBt4/detection_loss‚2.362198516726494 %loss/total_loss‚36.35640713099714 epoch‚9 $ _timestamp‚1.788446862919137e+09 0loss/FRt4/assignment_loss‚1.9246846890449525 1loss/SRqqt3/detection_loss‚2.6612113979127674 0loss/FBt1/detection_loss‚0.49969264218816534 _step‚898 0loss/FRt3/assignment_loss‚3.6303062387875147wðJIeÕ(ß 0loss/FRt1/assignment_loss‚1.2338943992342268 2loss/SRqqt2/assignment_loss‚1.3387822376719365 2loss/SRqqt3/assignment_loss‚0.9089231673904039 1loss/FBt2/assignment_loss‚0.24869223280984443 /loss/FRt4/detection_loss‚2.0891829046607016 0loss/SRqqt1/detection_loss‚0.904642808756658 2loss/SRqqt4/assignment_loss‚1.1943872924561478 1loss/FBt4/assignment_loss‚0.28494009931716846 1loss/SRqqt4/detection_loss‚1.6558421126433782 trainer/global_step‚10899 2loss/SRqqt1/assignment_loss‚0.6149466666193413 1loss/FBt3/assignment_loss‚0.05144746447331272 /loss/FBt3/detection_loss‚2.6165456250309944 1loss/SRqqt2/detection_loss‚1.6188871315370004 /loss/FRt2/assignment_loss‚1.790330994129181 /loss/FRt1/detection_loss‚1.8216173024404616 .loss/FBt2/detection_loss‚2.264351119597753 .loss/FBt4/detection_loss‚2.362198516726494 %loss/total_loss‚36.35640713099714 epoch‚9 $ _timestamp‚1.788446862919137e+09 0loss/FRt4/assignment_loss‚1.9246846890449525 1loss/SRqqt3/detection_loss‚2.6612113979127674 0loss/FBt1/detection_loss‚0.49969264218816534 _step‚898 0loss/FRt3/assignment_loss‚3.6303062387875147 1loss/FBt1/assignment_loss‚0.09856781850032974 /loss/FRt2/detection_loss‚2.1882120087742805 /loss/FRt3/detection_loss‚2.3541202602941276 _runtime‚9302.963727378‚9kß œÖ(‚(:‘ š‘æÔʬòÈ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:5qö×(‚(:® þ10i+š‘æÔÒª‡Ë disk.drbd1777.in‚0$ memory_percent‚77.13336944580078 cpu‚25.3406087853174 disk.sda3.in‚0.5625 disk.sda3.out‚ 341.8828125 disk.dm-1.out‚ 341.8828125 disk.rbd2.out‚ 649.517578125& proc.memory.availableMB‚ 3746.46875% disk./.usageGB‚1107.4736404418945 disk.sda.in‚0.5625 disk.sda1.out‚0* proc.memory.percent‚16.390228271484375 disk.rbd0.out‚0 network.sent‚ 2.7626995e+07 disk.drbd1777.out‚0 disk.sda.out‚ 341.8828125 proc.cpu.threads‚17& disk.nvme0n1.out‚81461.09130859375# disk.dm-0.out‚81461.09130859375 disk.rbd1.out‚ 4.662109375 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.21484375 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚81461.09130859375 disk.rbd2.in‚0 disk.sda1.in‚0 network.recv‚ 3.375516e+06 disk.dm-0.in‚ 1778.21484375 disk.rbd1.in‚0 proc.memory.rssMB‚2685.375) disk./.usagePercent‚74.32348524233305# disk.nvme0c0n1.in‚ 1778.214843757ˆŸØ(‚(:” ©‘æÔð¾¥É gpu.0.gpu‚70.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚261.392' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.278400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Û3Ù(‚(:¨ ©‘æÔØ—ôÊ disk.drbd1777.in‚0 disk.rbd1.out‚ 4.662109375 disk.rbd3.in‚0( proc.memory.availableMB‚ 3746.4609375 proc.cpu.threads‚17 disk.dm-1.in‚0.5625 disk.sda.out‚ 342.296875 disk.rbd3.out‚0# memory_percent‚77.1334171295166 proc.memory.rssMB‚2685.375 network.recv‚ 3.381459e+06 disk.dm-1.out‚ 342.296875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚81461.498046875 disk.sda3.in‚0.5625 disk.dm-0.in‚ 1778.21484375 network.sent‚ 2.7735981e+07 disk.sda3.out‚ 342.296875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.21484375* proc.memory.percent‚16.390228271484375 disk.sda.in‚0.5625 disk.sda2.in‚0& disk.nvme0c0n1.out‚81461.498046875) disk./.usagePercent‚74.32348140222003% disk./.usageGB‚1107.4735832214355 disk.rbd2.out‚ 651.216796875! disk.nvme0n1.in‚ 1778.21484375! disk.dm-0.out‚81461.498046875 cpu‚25.340117032491577 disk.sda1.in‚0 disk.rbd0.out‚0Œ,å2Ú(- _wandbruntime‚9341  _runtime‚9341½åoÉÛ(à _step‚899 ' lr-AdamW/pg1‚1.0419087450068975e-05 ' lr-AdamW/pg2‚1.0419087450068975e-05 trainer/global_step‚10949 $ _timestamp‚1.788446901246988e+09 _runtime‚9341.291669274…W"ýÎÜ(È _runtime‚9341.291669274 _step‚899 ' lr-AdamW/pg1‚1.0419087450068975e-05 ' lr-AdamW/pg2‚1.0419087450068975e-05 trainer/global_step‚10949 $ _timestamp‚1.788446901246988e+09ƒwípÿ2Ý(- _wandbruntime‚9341  _runtime‚9341Z'ë”#(Þ(P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204518652 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1244131063945728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1244133690546688 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204518652 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27836004495872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1204160166790144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 _runtime‚9341.292621697 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1046848267708416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20337250617856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76065098 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1244131063945728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1223796413537280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162068655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20300912748998 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1223796439928832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7884469012477283e+09 _step‚900 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567267 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204498618 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚27836048795136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105194832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105194835 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1074684244069888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚415 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181259933 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1074684316503552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204498618 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20334650408448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105194835 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1074684244069888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚415 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181259933 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1074684316503552 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚72433664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204498618 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20334650408448 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1244131063945728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1244133690546688 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204518652 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚27836004495872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1204160166790144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 _runtime‚9341.292621697 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1046848267708416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20337250617856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76065098 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1244131063945728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1223796413537280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162068655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20300912748998 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1223796439928832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7884469012477283e+09 _step‚900„Cg-Å2à(- _wandbruntime‚9342  _runtime‚9342Ïd‰>#á(§N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162082891 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20339082959872 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105204031 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27838587284480 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1244243462594048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599737 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76071462 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚901 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204516852 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20339082959872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181275490 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1244243462594048 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1046940627627520 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20336482737152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204516852 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1223904379634176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1074779170626048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1204266482536448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1223904379634176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1074779243046400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181275903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1046940655761920 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1204266507964416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162082885 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162082891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.æ¹wïfreed‚27838542998528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599737 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76071872 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1224569224628114 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20305340376542 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074Þâ‡2'â(¬N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1046940655761920 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1204266507964416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162082885 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162082891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27838542998528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599737 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76071872 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚72420352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1224569224628114 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20305340376542 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162082891 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20339082959872 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105204031 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27838587284480 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1244243462594048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366599737 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76071462 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚901 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204516852 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20339082959872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181275490 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1244243462594048 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚44285952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1046940627627520 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20336482737152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204516852 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1223904379634176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1074779170626048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1204266482536448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1223904379634176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1074779243046400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181275903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720…7´2ã(- _wandbruntime‚9342  _runtime‚9342ÓS¥Ê]ä(× _runtime‚9342.029163724 2loss/SRqqt4/assignment_loss‚0.9743036983327733 2loss/FBt2/assignment_loss‚0.008517550294830775 0loss/FBt4/assignment_loss‚0.6600728972698562 1loss/SRqqt3/detection_loss‚2.6648476767924523 /loss/FBt4/detection_loss‚3.6714303890864053 %loss/total_loss‚40.45554818655845 0loss/SRqqt1/detection_loss‚0.883547773502548 /loss/FRt2/assignment_loss‚2.148582085257485 1loss/FBt1/assignment_loss‚0.15330690347199252 /loss/FRt2/detection_loss‚2.3970112428069115 .loss/FRt4/detection_loss‚2.834784096106887 1loss/SRqqt4/detection_loss‚1.7949704471561643 trainer/global_step‚10949 0loss/FRt3/assignment_loss‚3.5748895092142954 2loss/SRqqt3/assignment_loss‚1.7053643340064633 1loss/FBt3/assignment_loss‚0.25456112974643474 1loss/SRqqt2/detection_loss‚1.5790491146299075 /loss/FBt3/detection_loss‚1.8081152066588402 .loss/FRt1/detection_loss‚1.339567698139165 .loss/FBt2/detection_loss‚3.870968242486318 _step‚902 0loss/FBt1/detection_loss‚0.44907041945153586 /loss/FRt1/assignment_loss‚0.817261105137212 /loss/FRt4/assignment_loss‚2.734517017379403 2loss/SRqqt1/assignment_loss‚0.7254441303795081 2loss/SRqqt2/assignment_loss‚0.9050395764972804 /loss/FRt3/detection_loss‚2.5003259427537867 $ _timestamp‚1.788446901984594e+09 epoch‚9ñ™0bå(Ü .loss/FRt1/detection_loss‚1.339567698139165 .loss/FBt2/detection_loss‚3.870968242486318 _step‚902 0loss/FBt1/detection_loss‚0.44907041945153586 /loss/FRt1/assignment_loss‚0.817261105137212 /loss/FRt4/assignment_loss‚2.734517017379403 2loss/SRqqt1/assignment_loss‚0.7254441303795081 2loss/SRqqt2/assignment_loss‚0.9050395764972804 /loss/FRt3/detection_loss‚2.5003259427537867 $ _timestamp‚1.788446901984594e+09 epoch‚9 _runtime‚9342.029163724 2loss/SRqqt4/assignment_loss‚0.9743036983327733 2loss/FBt2/assignment_loss‚0.008517550294830775 0loss/FBt4/assignment_loss‚0.6600728972698562 1loss/SRqqt3/detection_loss‚2.6648476767924523 /loss/FBt4/detection_loss‚3.6714303890864053 %loss/total_loss‚40.45554818655845 0loss/SRqqt1/detection_loss‚0.883547773502548 /loss/FRt2/assignment_loss‚2.148582085257485 1loss/FBt1/assignment_loss‚0.15330690347199252 /loss/FRt2/detection_loss‚2.3970112428069115 .loss/FRt4/detection_loss‚2.834784096106887 1loss/SRqqt4/detection_loss‚1.7949704471561643 trainer/global_step‚10949 0loss/FRt3/assignment_loss‚3.5748895092142954 2loss/SRqqt3/assignment_loss‚1.7053643340064633 1loss/FBt3/assignment_loss‚0.25456112974643474 1loss/SRqqt2/detection_loss‚1.5790491146299075 /loss/FBt3/detection_loss‚1.8081152066588402†£‰ÒŸæ(‚(:” ¸‘æÔËèþÈ gpu.0.gpu‚71.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚259.779' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.955799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒkÔS:ç(‚(:¯ ¸‘æÔ½Ø‚Ë disk.sda.out‚ 342.8515625 disk.dm-0.in‚ 1778.21484375% proc.memory.availableMB‚ 3745.6875 network.recv‚ 3.385857e+06) disk./.usagePercent‚74.32348140222003% disk./.usageGB‚1107.4735832214355 disk.rbd1.out‚ 4.662109375 network.sent‚ 2.7741807e+07 disk.drbd1777.in‚0 disk.rbd2.out‚ 651.232421875 disk.sda3.in‚0.5625 disk.sda3.out‚ 342.8515625 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1778.21484375 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0$ memory_percent‚77.13813781738281* proc.memory.percent‚16.390228271484375 proc.cpu.threads‚17& disk.nvme0n1.out‚81462.35986328125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1778.21484375( disk.nvme0c0n1.out‚81462.35986328125 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚81462.35986328125 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 342.8515625 cpu‚25.339652056276247 disk.rbd3.out‚0 proc.memory.rssMB‚2685.375Ù‰2.Ÿè(‚(:” Ç‘æÔ¹É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.412400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î$ K*é(‚(:Ÿ Ç‘æÔ“ŠÓÊ disk.dm-1.out‚ 343.359375* proc.memory.percent‚16.390228271484375 network.sent‚ 2.7851364e+07 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.21484375! disk.rbd2.out‚651.38134765625 disk.sda2.in‚0# disk.nvme0c0n1.out‚ 81464.609375 disk.dm-0.in‚ 1778.21484375# memory_percent‚77.1418571472168' proc.memory.availableMB‚ 3745.078125 disk.sda.in‚0.5625 proc.memory.rssMB‚2685.375 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1778.21484375 cpu‚25.339191317143285 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0.5625 network.recv‚ 3.392741e+06$ disk./.usageGB‚1107.305534362793 disk.rbd1.out‚ 4.662109375 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 81464.609375 disk.sda.out‚ 343.359375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0.5625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.31220350231415! disk.nvme0n1.out‚ 81464.609375 disk.sda3.out‚ 343.3593759ÞòÃ’ê(‚(:‡ Ö‘æÔ¢…“É gpu.0.gpu‚17.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚91.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚D>ë(‚(:³ Ö‘æÔÛ°ÙË disk.drbd1777.out‚0 disk.sda.out‚ 343.87890625 disk.rbd3.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚651.39697265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚81465.25927734375* proc.memory.percent‚16.390228271484375 disk.rbd1.out‚ 4.662109375 disk.rbd0.out‚0 disk.dm-1.in‚0.5625 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.21484375& disk.nvme0n1.out‚81465.25927734375 disk.dm-1.out‚ 343.87890625$ memory_percent‚77.14042663574219 proc.memory.rssMB‚2685.375 network.recv‚ 3.396429e+06) disk./.usagePercent‚74.31221118254017# disk.nvme0c0n1.in‚ 1778.21484375 disk.drbd1777.in‚0# disk.dm-0.out‚81465.25927734375 proc.cpu.threads‚17 network.sent‚ 2.7856581e+07 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda3.in‚0.5625$ disk./.usageGB‚1107.305648803711 disk.rbd3.out‚0 disk.sda3.out‚ 343.87890625 disk.dm-0.in‚ 1778.21484375% proc.memory.availableMB‚ 3745.3125 cpu‚25.338724092108695³éG‹2ì(- _wandbruntime‚9380  _runtime‚9380hUÄœÈí( trainer/global_step‚10999 % _timestamp‚1.7884469403383856e+09 _runtime‚9380.383103221 _step‚903 & lr-AdamW/pg1‚9.533341912169494e-06 & lr-AdamW/pg2‚9.533341912169494e-06÷µNÏÍî(Ç & lr-AdamW/pg1‚9.533341912169494e-06 & lr-AdamW/pg2‚9.533341912169494e-06 trainer/global_step‚10999 % _timestamp‚1.7884469403383856e+09 _runtime‚9380.383103221 _step‚903‡ÇžÞÁ2ï(- _wandbruntime‚9380  _runtime‚9380Ÿ`±#(ð(P UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105654632 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162780449 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1229872217262018 _runtime‚9380.383972679 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1209477225503744 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1249622306662912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 % _timestamp‚1.7884469403391526e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182056517 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105654635 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051467670383616 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1229193425196032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76401392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 trainer/global_step‚10999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205433704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1229193398804480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368214159 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76401882 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162780455 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20426281257472 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1249619680061952 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1249619680061952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1051467642249216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20394991758274 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20428881466880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1051467670383616 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1229193425196032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76401392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 trainer/global_step‚10999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205433704 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1229193398804480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatã²^?sMonitor.on_train_batch_start/allocation.all.allocated‚ 368214159 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76401882 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162780455 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20426281257472 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1249619680061952 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1249619680061952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1051467642249216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20394991758274 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20428881466880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1229872217262018 _runtime‚9380.383972679 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1209477225503744 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1249622306662912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 % _timestamp‚1.7884469403391526e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182056517 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105654635 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1229301338509824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27964707358720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1229301364901376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1051560058437120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162794691 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20430713808896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105663828 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162794685 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205451964 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1079524689181184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162794691 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20396821101150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1249732078710272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1209583541250048 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1249729452096000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182071808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051560030302720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20430713808896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1249732078710272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205431904 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1079524765795840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9381.127799343 trainer/global_step‚10999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1229977764066322 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76408437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 $ _timestamp‚1.788446941082699e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1229301364901376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚27964658878464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20428113586176 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20428113586176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20394222816274 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246655 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1209583566678016 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182072268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105663831 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1229980387779166 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182072268 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105663831 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1229980387779166 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1229301338509824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚27964707358720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1229301364901376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1051560058437120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162794691 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20430713808896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105663828 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162794685 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205451964 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1079524689181184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162794691 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20396821101150 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1249732078710272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1209583541250048 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1249729452096000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182071808 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051560030302720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20430713808896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1249732078710272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205431904 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1079524765795840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9381.127799343 trainer/global_step‚10999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1229977764066322 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400‰|'/2õ(- _wandbruntime‚9381  _runtime‚9381Fàø[ö(Õ 0loss/FRt3/assignment_loss‚2.7783259169922934 0loss/FRt1/assignment_loss‚0.9827728180781655 1loss/SRqqt3/assignment_loss‚1.130810891777011 1loss/SRqqt4/assignment_loss‚1.232715377385946 .loss/FRt4/detection_loss‚2.378598896165689 2loss/SRqqt1/assignment_loss‚0.9403197853596973 /loss/FBt3/detection_loss‚1.7528757197516305 _step‚906 3loss/FBt2/assignment_loss‚0.0019071315182372928 /loss/FRt1/detection_loss‚1.6974166033222624 .loss/FRt3/detection_loss‚2.458712158648672 /loss/SRqqt3/detection_loss‚2.35525160495724 .loss/FBt2/detection_loss‚4.635289967060089 1loss/FBt3/assignment_loss‚0.05375319991019621 1loss/FBt4/assignment_loss‚0.09720239868329372 1loss/SRqqt2/detection_loss‚1.6887370793550622 $ _timestamp‚1.788446941084385e+09 0loss/FRt2/assignment_loss‚1.7858948938700618 /loss/FBt4/detection_loss‚2.3795029322306314 _runtime‚9381.128985428 /loss/FRt4/assignment_loss‚2.100053362903141 0loss/FBt1/assignment_loss‚0.1117337218031019 .loss/FRt2/detection_loss‚2.076878915940012 1loss/SRqqt1/detection_loss‚0.8578411533695752 1loss/SRqqt4/detection_loss‚1.7758174097125834 epoch‚9 2loss/SRqqt2/assignment_loss‚1.0148157838038077 /loss/FBt1/detection_loss‚0.5638491625701668 %loss/total_loss‚36.85107688516857 trainer/global_step‚10999¸®Gé`÷(Ú 0loss/FRt2/assignment_loss‚1.7858948938700618 /loss/FBt4/detection_loss‚2.3795029322306314 _runtime‚9381.128985428 /loss/FRt4/assignment_loss‚2.100053362903141 0loss/FBt1/assignment_loss‚0.1117337218031019 .loss/FRt2/detection_loss‚2.076878915940012 1loss/SRqqt1/detection_loss‚0.8578411533695752 1loss/SRqqt4/detection_loss‚1.7758174097125834 epoch‚9 2loss/SRqqt2/assignment_loss‚1.0148157838038077 /loss/FBt1/detection_loss‚0.5638491625701668 %loss/total_loss‚36.85107688516857 trainer/global_step‚10999 0loss/FRt3/assignment_loss‚2.7783259169922934 0loss/FRt1/assignment_loss‚0.9827728180781655 1loss/SRqqt3/assignment_loss‚1.130810891777011 1loss/SRqqt4/assignment_loss‚1.232715377385946 .loss/FRt4/detection_loss‚2.378598896165689 2loss/SRqqt1/assignment_loss‚0.9403197853596973 /loss/FBt3/detection_loss‚1.7528757197516305 _step‚906 3loss/FBt2/assignment_loss‚0.0019071315182372928 /loss/FRt1/detection_loss‚1.6974166033222624 .loss/FRt3/detection_loss‚2.458712158648672 /loss/SRqqt3/detection_loss‚2.35525160495724 .loss/FBt2/detection_loss‚4.635289967060089 1loss/FBt3/assignment_loss‚0.05375319991019621 1loss/FBt4/assignment_loss‚0.09720239868329372 1loss/SRqqt2/detection_loss‚1.6887370793550622 $ _timestamp‚1.788446941084385e+09ŠRÈ1‘žø(‚(:“ å‘æÔ…§úÈ gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚270.244' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.04880000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}4O<>ù(‚(:³ å‘æÔ”ÀÚÊ$ memory_percent‚77.14111804962158 proc.memory.rssMB‚2685.375 cpu‚25.338269668250465# disk.nvme0c0n1.in‚ 1778.21484375 disk.sda2.in‚0! disk.nvme0n1.in‚ 1778.21484375 proc.cpu.threads‚17# disk./.usageGB‚1107.42924118042 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.390228271484375 disk.dm-0.in‚ 1778.21484375 disk.sda3.out‚ 344.4453125 disk.dm-1.in‚0.5625 disk.rbd1.out‚ 4.662109375 disk.sda1.in‚0( disk.nvme0c0n1.out‚81592.27880859375 disk.drbd1777.out‚0! disk.rbd2.out‚651.46728515625 disk.rbd3.out‚0& disk.nvme0n1.out‚81592.27880859375) proc.memory.availableMB‚ 3745.19921875 network.recv‚ 3.400656e+06# disk.dm-0.out‚81592.27880859375 disk.rbd3.in‚0 network.sent‚ 2.7862527e+07) disk./.usagePercent‚74.32050557064203 disk.dm-1.out‚ 344.4453125 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 344.4453125 disk.sda3.in‚0.5625 disk.rbd1.in‚0 disk.sda2.out‚0ú"ù”ú(‚(:‰ ô‘æÔê°‰É gpu.0.gpu‚60.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚102.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÑ8û(‚(:­ ô‘æÔˆˆûË disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.21484375 disk.sda.in‚0.5625 proc.memory.rssMB‚2685.375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1778.21484375 disk.rbd1.out‚ 4.662109375! disk.rbd2.out‚653.05322265625 disk.sda2.in‚0 disk.drbd1777.in‚0 network.sent‚ 2.7971508e+07 network.recv‚ 3.407363e+06! disk.dm-0.out‚81719.408203125' proc.memory.availableMB‚ 986.7109375 cpu‚25.337777723575215 disk.dm-0.in‚ 1778.21484375 disk.rbd2.in‚0% disk./.usageGB‚1107.5533332824707 disk.rbd3.out‚0$ disk.nvme0n1.out‚81719.408203125 disk.sda3.in‚0.5625& disk.nvme0c0n1.out‚81719.408203125 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 345.1171875 disk.rbd1.in‚0) disk./.usagePercent‚74.32883349573086 disk.sda3.out‚ 345.1171875 disk.sda.out‚ 345.1171875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.97759437561035* proc.memory.percent‚16.390228271484375}ªRùIü(‚(:“ ƒ’æÔÙ¡É gpu.0.gpu‚72.0 gpu.0.memory‚13, gpuıxHU.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.58' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.916000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(~ 71ý(‚(:¦ ƒ’æÔÐóúË disk.sda2.in‚0 cpu‚25.337320776852067! disk.dm-0.out‚81721.251953125 disk.drbd1777.in‚0! disk.rbd2.out‚653.19384765625$ memory_percent‚93.97916793823242 disk.nvme0n1.in‚ 1778.21875 disk.drbd1777.out‚0 disk.sda.in‚0.5625& proc.memory.availableMB‚ 986.453125% disk./.usageGB‚1107.5545043945312 disk.sda3.in‚0.5625 disk.sda1.out‚0 disk.rbd3.in‚0 network.recv‚ 3.412105e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 1778.21875 disk.dm-1.in‚0.5625 disk.rbd1.in‚0 disk.sda.out‚ 345.73828125 disk.rbd3.out‚0) disk./.usagePercent‚74.32891209004387$ disk.nvme0n1.out‚81721.251953125& disk.nvme0c0n1.out‚81721.251953125 disk.dm-0.in‚ 1778.21875 disk.rbd1.out‚ 4.662109375 disk.sda2.out‚0 proc.memory.rssMB‚2685.375 disk.sda3.out‚ 345.73828125 disk.rbd0.in‚0 disk.dm-1.out‚ 345.73828125* proc.memory.percent‚16.390228271484375 proc.cpu.threads‚17 network.sent‚ 2.7977851e+07 disk.rbd2.in‚0 disk.sda1.in‚0í¥v2þ(- _wandbruntime‚9420  _runtime‚9420 ËNÇÿ(Á trainer/global_step‚11049 % _timestamp‚1.7884469802240686e+09 _runtime‚ 9420.26875376 _step‚907 & lr-AdamW/pg1‚8.685988318294761e-06 & lr-AdamW/pg2‚8.685988318294761e-06\ìÎàÌ€)Æ % _timestamp‚1.7884469802240686e+09 _runtime‚ 9420.26875376 _step‚907 & lr-AdamW/pg1‚8.685988318294761e-06 & lr-AdamW/pg2‚8.685988318294761e-06 trainer/global_step‚11049‹%ü—v2)- _wandbruntime‚9420  _runtime‚9420þL½#(‚)P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1255110922779136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1255108296178176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20520512315904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206348756 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28088197270016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1255110922779136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28088148776448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182851128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884469802249134e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163492255 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1235280756700226 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1235278132987590 _runtime‚9420.269961335 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1234590384071680 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1234590384071680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163492249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1255108296178176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76736693 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1234590410463232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206328722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1214794258789376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚908 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944‡½DË((ƒ)¢P H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1255110922779136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1255108296178176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20520512315904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206348756 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28088197270016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1255110922779136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28088148776448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182851128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884469802249134e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163492255 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1235280756700226 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1235278132987590 _runtime‚9420.269961335 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1234590384071680 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1234590384071680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163492249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1255108296178176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76736693 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1234590410463232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206328722 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1214794258789376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚908Œin\»2„)- _wandbruntime‚9421  _runtime‚9421W HY+'…)¥N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206367016 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1234698350168576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206346956 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182866471 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163506491 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20522344657920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1056179461112320 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1235388927217374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20485703540882 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163506485 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1234698350168576 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20522344657920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206367016 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056179432977920 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 trainer/global_step‚11049 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234698323777024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163506491 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20519744435200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20519744435200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76743290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1255218068212224 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1084270111024128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚909 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1255220694826496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873507 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1255220694826496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20488301825758 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1234698323777024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106123631 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28090678046208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1235386303504530ݤ¬£Ž†)ªN LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.ÏJ|ñ¢&large_pool.peak‚ 17729555456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1255218068212224 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1084270111024128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚909 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1255220694826496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873507 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1255220694826496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20488301825758 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1234698323777024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106123631 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28090678046208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1235386303504530 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206367016 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1234698350168576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206346956 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182866471 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163506491 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20522344657920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1056179461112320 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1235388927217374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20485703540882 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163506485 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1234698350168576 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20522344657920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206367016 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1056179432977920 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 trainer/global_step‚11049 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234698323777024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163506491 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20519744435200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20519744435200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76743290Åñ2‡)- _wandbruntime‚9421  _runtime‚9421ù9#_ˆ)Ù 2loss/SRqqt1/assignment_loss‚0.9033483345929074 2loss/SRqqt4/assignment_loss‚1.1435088745604913 .loss/FRt4/detection_loss‚2.252020124346018 /loss/FBt1/detection_loss‚0.6389437228374716 epoch‚9 /loss/FRt1/assignment_loss‚1.096812859469769 0loss/FRt3/assignment_loss‚3.0252048608868622 1loss/FBt3/assignment_loss‚0.20625677495263517 trainer/global_step‚11049 _step‚910 0loss/FRt4/assignment_loss‚1.9165677333419973 /loss/FRt3/detection_loss‚2.4916806311385846 0loss/SRqqt3/detection_loss‚2.118178271149334 1loss/SRqqt4/detection_loss‚1.7747229723116527 .loss/FBt3/detection_loss‚2.582659681638082 /loss/FRt1/detection_loss‚1.5291451033496974 0loss/FRt2/assignment_loss‚2.2755084013938904 2loss/SRqqt3/assignment_loss‚1.3575552628424608 2loss/FBt4/assignment_loss‚0.030845741295446975 1loss/FBt1/assignment_loss‚0.10974593264757042 /loss/FBt4/detection_loss‚2.5498161230768477 .loss/FRt2/detection_loss‚2.177718851566315 0loss/FBt2/assignment_loss‚0.4699658693385903 $ _timestamp‚1.788446980974025e+09 _runtime‚9421.018669148 1loss/SRqqt1/detection_loss‚0.7401870438344267 /loss/FBt2/detection_loss‚1.5263180461796848 2loss/SRqqt2/assignment_loss‚1.0855247043073177 1loss/SRqqt2/detection_loss‚1.7326470537483691 &loss/total_loss‚35.734882974806425Á¢C‹d‰)Þ 2loss/SRqqt1/assignment_loss‚0.9033483345929074 2loss/SRqqt4/assignment_loss‚1.1435088745604913 .loss/FRt4/detection_loss‚2.252020124346018 /loss/FBt1/detection_loss‚0.6389437228374716 epoch‚9 /loss/FRt1/assignment_loss‚1.096812859469769 0loss/FRt3/assignment_loss‚3.0252048608868622 1loss/FBt3/assignment_loss‚0.20625677495263517 trainer/global_step‚11049 _step‚910 0loss/FRt4/assignment_loss‚1.9165677333419973 /loss/FRt3/detection_loss‚2.4916806311385846 0loss/SRqqt3/detection_loss‚2.118178271149334 1loss/SRqqt4/detection_loss‚1.7747229723116527 .loss/FBt3/detection_loss‚2.582659681638082 /loss/FRt1/detection_loss‚1.5291451033496974 0loss/FRt2/assignment_loss‚2.2755084013938904 2loss/SRqqt3/assignment_loss‚1.3575552628424608 2loss/FBt4/assignment_loss‚0.030845741295446975 1loss/FBt1/assignment_loss‚0.10974593264757042 /loss/FBt4/detection_loss‚2.5498161230768477 .loss/FRt2/detection_loss‚2.177718851566315 0loss/FBt2/assignment_loss‚0.4699658693385903 $ _timestamp‚1.788446980974025e+09 _runtime‚9421.018669148 1loss/SRqqt1/detection_loss‚0.7401870438344267 /loss/FBt2/detection_loss‚1.5263180461796848 2loss/SRqqt2/assignment_loss‚1.0855247043073177 1loss/SRqqt2/detection_loss‚1.7326470537483691 &loss/total_loss‚35.734882974806425Ž(ÙIŸŠ)‚(:” ’’æÔŠ·˜É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚167.443' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.488600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ñ?o*‹)‚(:Ÿ ’’æÔ¯¥ØË disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd2.in‚0# memory_percent‚93.9786434173584# disk.dm-0.out‚81722.51025390625 disk.sda1.out‚0( disk.nvme0c0n1.out‚81722.51025390625 disk.dm-0.in‚ 1778.21875) disk./.usagePercent‚74.32895484330209 disk.sda3.out‚346.1875! disk.rbd2.out‚653.20947265625 disk.sda.in‚0.5625 cpu‚25.336866776193993 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 1778.21875 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.dm-1.out‚346.1875 disk.rbd1.out‚ 4.662109375 disk.sda2.out‚0 proc.memory.rssMB‚2685.375 network.recv‚ 3.41587e+06% disk./.usageGB‚1107.5551414489746 disk.drbd1777.out‚0 disk.sda.out‚346.1875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1778.21875& disk.nvme0n1.out‚81722.51025390625 network.sent‚ 2.7983055e+07 disk.rbd1.in‚0' proc.memory.availableMB‚ 986.5390625* proc.memory.percent‚16.390228271484375&µøŸŒ)‚(:” ¡’æÔß­éÈ gpu.0.gpu‚83.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚45.0 gpu.0.powerWatts‚136.892' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.378399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óîAÁ8)‚(:­ ¡’æÔÙË disk.sda2.in‚0$ memory_percent‚93.98057460784912 proc.cpu.threads‚17) disk./.usagePercent‚74.32895100318908% disk./.usageGB‚1107.5550842285156 disk.rbd2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚81722.84814453125 disk.rbd1.out‚ 4.662109375( proc.memory.availableMB‚ 986.22265625 proc.memory.rssMB‚2685.375 cpu‚25.336386387311514 disk.nvme0n1.in‚ 1778.21875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚81722.84814453125 disk.drbd1777.in‚0 disk.rbd2.out‚653.2958984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 1778.21875 disk.dm-1.out‚ 346.66796875 network.recv‚ 3.422771e+06& disk.nvme0n1.out‚81722.84814453125 disk.sda3.out‚ 346.66796875 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.sent‚ 2.8092073e+07 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 1778.21875 disk.sda.in‚0.5625 disk.sda.out‚ 346.66796875 disk.rbd3.in‚0 disk.sda3.in‚0.5625 disk.dm-1.in‚0.5625* proc.memory.percent‚16.390228271484375Üj2Ž)- _wandbruntime‚9459  _runtime‚9459Åo±¤Ç)Á & lr-AdamW/pg1‚7.877203255663696e-06 & lr-AdamW/pg2‚7.877203255663696e-06 trainer/global_step‚11099 $ _timestamp‚1.788447019289022e+09 _runtime‚9459.333716116 _step‚911CCaÌ)Æ _runtime‚9459.333716116 _step‚911 & lr-AdamW/pg1‚7.877203255663696e-06 & lr-AdamW/pg2‚7.877203255663696e-06 trainer/global_step‚11099 $ _timestamp‚1.788447019289022e+09—eîI2‘)- _wandbruntime‚9459  _runtime‚9459•)Ú€#(’)P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371447823 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371467863 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28214309390336 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1239987369338880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164204049 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207243774 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1260596912294400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77071018 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1260599538895360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20577953207490 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164204055 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1260599538895360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20612143164928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1260596912294400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20612143164928 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1060706475734016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884470192897837e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207263808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77071461 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28214262993920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207243774 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164204055 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1220111342930944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20575354922822 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 trainer/global_step‚11099 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371467863 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28214309390336 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1239987369338880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164204049 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 =6DeviceStatsMoniW’’Ûtor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207243774 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1260596912294400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77071018 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1260599538895360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20577953207490 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164204055 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1260599538895360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20612143164928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1260596912294400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20612143164928 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1060706475734016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884470192897837e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400{xG2”)- _wandbruntime‚9460  _runtime‚9460+5Š+'•)¥N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1089015705955328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28216795784704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 trainer/global_step‚11099 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1260706684328448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1240095309044224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20577184265490 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500359 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480293 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1240095335435776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207282068 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106583428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1220217684105216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1060798863787520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28216842167808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1260709310942720 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1260709310942720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1240797466655582 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20611375284224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20613975506944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183661057 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1240095335435776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20613975506944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207262008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376!f) 0'–)ªN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1089015705955328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28216795784704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 trainer/global_step‚11099 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1260706684328448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1240095309044224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20577184265490 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500359 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480293 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1240095335435776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207282068 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106583428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚46383104 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1220217684105216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1060798863787520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28216842167808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1260709310942720 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1260709310942720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1240797466655582 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20611375284224 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20613975506944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183661057 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1240095335435776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20613975506944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207262008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0‘ 2—)- _wandbruntime‚9460  _runtime‚9460a ÏúZ˜)Ô 2loss/SRqqt2/assignment_loss‚1.3851908013891137 2loss/SRqqt4/assignment_loss‚1.1603094463142851 /loss/FRt1/detection_loss‚1.4839234049332903 /loss/FBt3/detection_loss‚2.3755016550421715 /loss/FRt4/detection_loss‚2.2932275346503026 /loss/FBt4/detection_loss‚2.7870777249336243 trainer/global_step‚11099 .loss/FRt1/assignment_loss‚0.67372742231856 /loss/FRt2/assignment_loss‚2.852839006483555 2loss/SRqqt1/assignment_loss‚0.8998411308015961 /loss/FBt4/assignment_loss‚0.206427383585833 _runtime‚9460.080045754 _step‚914 .loss/FRt3/assignment_loss‚2.53876255364979 1loss/FBt3/assignment_loss‚0.19331961768330075 /loss/FRt2/detection_loss‚2.4118779115378857 0loss/SRqqt4/detection_loss‚1.837111465798484 /loss/FBt2/detection_loss‚2.5971842954556146 epoch‚9 # _timestamp‚1.78844702003547e+09 1loss/SRqqt2/detection_loss‚1.7312032562844895 /loss/FBt1/detection_loss‚0.4689005538929514 0loss/FRt4/assignment_loss‚2.5839701963634027 2loss/SRqqt3/assignment_loss‚1.4026593938469887 1loss/FBt1/assignment_loss‚0.15472533885954204 1loss/FBt2/assignment_loss‚0.07613097350379879 /loss/FRt3/detection_loss‚2.4822772135724818 1loss/SRqqt1/detection_loss‚0.8061723913997412 &loss/total_loss‚37.615949649772176 0loss/SRqqt3/detection_loss‚2.213588977471376œP, _™)Ù 1loss/SRqqt2/detection_loss‚1.7312032562844895 /loss/FBt1/detection_loss‚0.4689005538929514 0loss/FRt4/assignment_loss‚2.5839701963634027 2loss/SRqqt3/assignment_loss‚1.4026593938469887 1loss/FBt1/assignment_loss‚0.15472533885954204 1loss/FBt2/assignment_loss‚0.07613097350379879 /loss/FRt3/detection_loss‚2.4822772135724818 1loss/SRqqt1/detection_loss‚0.8061723913997412 &loss/total_loss‚37.615949649772176 0loss/SRqqt3/detection_loss‚2.213588977471376 2loss/SRqqt2/assignment_loss‚1.3851908013891137 2loss/SRqqt4/assignment_loss‚1.1603094463142851 /loss/FRt1/detection_loss‚1.4839234049332903 /loss/FBt3/detection_loss‚2.3755016550421715 /loss/FRt4/detection_loss‚2.2932275346503026 /loss/FBt4/detection_loss‚2.7870777249336243 trainer/global_step‚11099 .loss/FRt1/assignment_loss‚0.67372742231856 /loss/FRt2/assignment_loss‚2.852839006483555 2loss/SRqqt1/assignment_loss‚0.8998411308015961 /loss/FBt4/assignment_loss‚0.206427383585833 _runtime‚9460.080045754 _step‚914 .loss/FRt3/assignment_loss‚2.53876255364979 1loss/FBt3/assignment_loss‚0.19331961768330075 /loss/FRt2/detection_loss‚2.4118779115378857 0loss/SRqqt4/detection_loss‚1.837111465798484 /loss/FBt2/detection_loss‚2.5971842954556146 epoch‚9 # _timestamp‚1.78844702003547e+09’󴔚)‚(:‰ °’æÔö€ÿÊ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚147.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.5862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜Ê/Ë›)‚(:Œ °’æÔ“˜›Í disk.rbd3.in‚0 disk.sda3.out‚ 347.01953125 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.out‚81731.75% disk./.usageGB‚1107.5630645751953 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚653.3115234375 disk.sda.in‚0.5625 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚81731.75 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 disk.dm-1.out‚ 347.01953125 proc.cpu.threads‚17 network.sent‚ 2.8097887e+07 disk.sda3.in‚0.5625 disk.rbd0.in‚0# memory_percent‚93.9791202545166 proc.memory.rssMB‚2685.375 cpu‚25.33595165888202( disk./.usagePercent‚74.3294865709507 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.out‚81731.75 disk.nvme0c0n1.in‚ 1778.21875 disk.dm-0.in‚ 1778.21875 disk.nvme0n1.in‚ 1778.21875' proc.memory.availableMB‚ 986.4609375* proc.memory.percent‚16.390228271484375 network.recv‚ 3.4272e+06 disk.sda.out‚ 347.019531258­™·“œ)‚(:ˆ ¿’æÔÔ¨³É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚97.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Åž+)‚(:  ¿’æÔÏËÌ disk.dm-0.in‚ 1778.21875 disk.sda1.in‚0 disk.rbd1.out‚ 4.662109375 disk.rbd2.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 347.6640625 network.recv‚ 3.433646e+06 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚93.97637844085693* proc.memory.percent‚16.390228271484375 disk.sda.in‚0.5625 disk.sda.out‚ 347.6640625) disk./.usagePercent‚74.32945533803152$ disk./.usageGB‚1107.562599182129 disk.rbd2.out‚653.4365234375 disk.sda2.in‚0 disk.nvme0n1.in‚ 1778.21875 disk.dm-1.in‚0.5625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.out‚81732.05078125 disk.rbd1.in‚0# disk.nvme0n1.out‚81732.05078125 disk.nvme0c0n1.in‚ 1778.21875 proc.cpu.threads‚17 disk.rbd3.out‚0( proc.memory.availableMB‚ 986.91015625 proc.memory.rssMB‚2685.375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 347.6640625 network.sent‚ 2.8207197e+07 disk.rbd0.in‚0% disk.nvme0c0n1.out‚81732.05078125 cpu‚25.335480295911015¹É”ž)‚(:‰ Î’æÔúËëÈ gpu.0.gpu‚71.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚266.772' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í!¡9Ÿ)‚(:® Î’æÔÙƒÑË disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0.5625 disk.rbd0.out‚0 disk.dm-1.out‚ 347.953125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 disk.rbd2.out‚655.0107421875 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.335033245689733 network.recv‚ 3.437536e+06 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1778.25390625( disk.nvme0c0n1.out‚81732.48095703125 proc.cpu.threads‚17$ disk./.usageGB‚1107.562656402588 disk.sda2.out‚0& disk.nvme0n1.out‚81732.48095703125) disk./.usagePercent‚74.32945917814455 disk.drbd1777.in‚0 disk.sda.out‚ 347.953125 disk.rbd3.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 986.1640625* proc.memory.percent‚16.390228271484375 network.sent‚ 2.8212516e+07 disk.sda3.out‚ 347.953125 disk.dm-0.in‚ 1778.25390625$ memory_percent‚93.98093223571777 proc.memory.rssMB‚2685.375# disk.nvme0c0n1.in‚ 1778.25390625# disk.dm-0.out‚81732.48095703125 disk.dm-1.in‚0.5625SaÙ=2 )- _wandbruntime‚9498  _runtime‚9498 OÈ¡) _step‚915 & lr-AdamW/pg1‚7.107155273871113e-06 & lr-AdamW/pg2‚7.107155273871113e-06 trainer/global_step‚11149 % _timestamp‚1.7884470587603972e+09 _runtime‚9498.805057347–ºo"Í¢)Ç trainer/global_step‚11149 % _timestamp‚1.7884470587603972e+09 _runtime‚9498.805057347 _step‚915 & lr-AdamW/pg1‚7.107155273871113e-06 & lr-AdamW/pg2‚7.107155273871113e-06“'@w2£)- _wandbruntime‚9498  _runtime‚9498)jÅƤ)œP KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164915855 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1245384354606080 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1093665341409792 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 trainer/global_step‚11149 NFDeviceStatsMonitor.on_train_batch_start€8¨ "%/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20701173804544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚916 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164915849 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373074675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1225428376216576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20703774013952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20666835647430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164915855 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208178860 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77405542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20669433932098 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20703774013952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77406004 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28339491134976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107034032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28339541725696 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1266085528410624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1065325850274816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208178860 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚05SU'(¥)¡P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1245384380997632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1266085528410624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1246097835576642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 _runtime‚ 9498.80603562 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208158826 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1266085528410624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1065325850274816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208178860 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164915855 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1245384354606080 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1093665341409792 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 trainer/global_step‚11149 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20701173804544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚916 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164915849 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373074675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1225428376216576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20703774013952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20666835647430 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164915855 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208178860 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77405542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20669433932098 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20703774013952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77406004 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28339491134976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107034032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28339541725696 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208177060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20705606355968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20668664990098 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1266195300444672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184455348 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77412572 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1093760238228480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28342050477568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20703006133248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 _step‚917 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107145 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1246203382380946 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884470595068166e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225534742818816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 373127211 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1245492294311424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164930085 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164930091 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1245492320702976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚9499.552653365 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1245492294311424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20705606355968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208197120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20703006133248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164930085 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1246206006093790 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77412120 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1065418266462720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208197120 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1266197927058944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20671263274974 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÁŸ,Ç5 ¨)¬N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20703006133248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164930085 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1246206006093790 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77412120 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1065418266462720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208197120 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1266197927058944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20671263274974 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208177060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20705606355968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 OGDeviceStatsMonitor.on_tôù¡œýrain_batch_end/inactive_split.small_pool.current‚452 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20668664990098 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1266195300444672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184455348 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77412572 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1093760238228480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28342050477568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20703006133248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 _step‚917 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 373107145 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1246203382380946 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 % _timestamp‚1.7884470595068166e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225534742818816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 373127211 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1245492294311424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164930085 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164930091 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1245492320702976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 _runtime‚9499.552653365 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1245492294311424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20705606355968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208197120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0•Ž(2©)- _wandbruntime‚9499  _runtime‚9499º^Äh\ª)Ö 1loss/SRqqt1/detection_loss‚1.0377188924279424 1loss/SRqqt4/detection_loss‚1.5526279642113618 .loss/FBt2/detection_loss‚1.996169838038358 _step‚918 2loss/SRqqt1/assignment_loss‚0.7852224094507795 epoch‚9 0loss/FRt2/assignment_loss‚1.8100140618114937 /loss/FBt4/detection_loss‚2.5388739109039307 %loss/total_loss‚36.63615677588977 1loss/FBt2/assignment_loss‚0.14816912765011445 /loss/FRt4/detection_loss‚2.4873948149383067 .loss/FRt1/detection_loss‚1.738852766384794 .loss/FRt3/detection_loss‚2.566270903251567 1loss/SRqqt2/detection_loss‚1.4643389213597402 .loss/FBt3/detection_loss‚2.501159317791462 trainer/global_step‚11149 % _timestamp‚1.7884470595089316e+09 _runtime‚ 9499.55378677 /loss/FRt3/assignment_loss‚3.064433024610792 0loss/FRt4/assignment_loss‚2.0608313620090484 1loss/FBt1/assignment_loss‚0.05856996083550708 /loss/FBt1/detection_loss‚0.6229924873022556 0loss/FRt1/assignment_loss‚1.4524540413509717 2loss/SRqqt3/assignment_loss‚1.4734480642911159 0loss/FBt3/assignment_loss‚0.1735917308687931 1loss/FBt4/assignment_loss‚0.11651870141213294 1loss/SRqqt3/detection_loss‚2.2414027551809945 2loss/SRqqt2/assignment_loss‚1.0380403802264482 2loss/SRqqt4/assignment_loss‚1.2176040223400508 .loss/FRt2/detection_loss‚2.489457317241808T¹¥2a«)Û 0loss/FRt2/assignment_loss‚1.8100140618114937 /loss/FBt4/detection_loss‚2.5388739109039307 %loss/total_loss‚36.63615677588977 1loss/FBt2/assignment_loss‚0.14816912765011445 /loss/FRt4/detection_loss‚2.4873948149383067 .loss/FRt1/detection_loss‚1.738852766384794 .loss/FRt3/detection_loss‚2.566270903251567 1loss/SRqqt2/detection_loss‚1.4643389213597402 .loss/FBt3/detection_loss‚2.501159317791462 trainer/global_step‚11149 % _timestamp‚1.7884470595089316e+09 _runtime‚ 9499.55378677 /loss/FRt3/assignment_loss‚3.064433024610792 0loss/FRt4/assignment_loss‚2.0608313620090484 1loss/FBt1/assignment_loss‚0.05856996083550708 /loss/FBt1/detection_loss‚0.6229924873022556 0loss/FRt1/assignment_loss‚1.4524540413509717 2loss/SRqqt3/assignment_loss‚1.4734480642911159 0loss/FBt3/assignment_loss‚0.1735917308687931 1loss/FBt4/assignment_loss‚0.11651870141213294 1loss/SRqqt3/detection_loss‚2.2414027551809945 2loss/SRqqt2/assignment_loss‚1.0380403802264482 2loss/SRqqt4/assignment_loss‚1.2176040223400508 .loss/FRt2/detection_loss‚2.489457317241808 1loss/SRqqt1/detection_loss‚1.0377188924279424 1loss/SRqqt4/detection_loss‚1.5526279642113618 .loss/FBt2/detection_loss‚1.996169838038358 _step‚918 2loss/SRqqt1/assignment_loss‚0.7852224094507795 epoch‚9–îS9‹Ÿ¬)‚(:” Ý’æÔê´„É gpu.0.gpu‚72.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚263.275' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.654999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Ïï˜=­)‚(:² Ý’æÔǧìÊ disk.rbd2.out‚655.1513671875 disk.sda.in‚0.5625 disk.sda2.in‚0 network.sent‚ 2.8218519e+07 disk.dm-0.in‚ 1778.25390625 disk.drbd1777.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚81733.32470703125 disk.rbd0.out‚0$ memory_percent‚93.98140907287598# disk.dm-0.out‚81733.32470703125 disk.rbd0.in‚0 proc.memory.rssMB‚2685.375* proc.memory.percent‚16.390228271484375 cpu‚25.3345892659364 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚81733.32470703125 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1778.25390625) disk./.usagePercent‚74.32962916714729 disk.sda1.in‚0! disk.nvme0n1.in‚ 1778.25390625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0.5625 disk.sda3.out‚ 348.71484375' proc.memory.availableMB‚ 986.0859375 disk.rbd1.out‚ 4.748046875 network.recv‚ 3.44228e+06% disk./.usageGB‚1107.5651893615723 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 348.71484375 disk.sda.out‚ 348.71484375ŠHÙž®)‚(:“ ì’æÔ„üúÈ gpu.0.gpu‚59.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚262.744' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.54880000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘hÈ2?¯)‚(:´ ì’æÔ•±ñË disk.drbd1777.out‚0 disk.rbd1.out‚ 4.810546875 disk.sda3.in‚0.5625 disk.rbd0.out‚0) disk./.usagePercent‚74.32962225494386 disk.dm-1.out‚ 349.08203125 disk.rbd2.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚81735.14599609375 disk.rbd0.in‚0$ disk./.usageGB‚1107.565086364746 disk.rbd2.out‚655.1513671875 disk.sda1.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.out‚ 349.08203125# disk.nvme0c0n1.in‚ 1778.26171875# disk.dm-0.out‚81735.14599609375 disk.sda2.out‚0$ memory_percent‚93.97995471954346( proc.memory.availableMB‚ 986.32421875 network.recv‚ 3.44903e+06( disk.nvme0c0n1.out‚81735.14599609375 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.in‚0 disk.sda.in‚0.5625 disk.sda.out‚ 349.08203125 disk.sda2.in‚0 disk.rbd3.in‚0 network.sent‚ 2.8327461e+07 disk.dm-1.in‚0.5625 disk.rbd3.out‚0 proc.memory.rssMB‚2685.375* proc.memory.percent‚16.390228271484375 cpu‚25.334136740622654 proc.cpu.threads‚17 disk.rbd1.in‚0XSUˆ2°)- _wandbruntime‚9538  _runtime‚9538ä25ȱ) % _timestamp‚1.7884470979921188e+09 _runtime‚9538.036799586 _step‚919 & lr-AdamW/pg1‚6.376004849762261e-06 & lr-AdamW/pg2‚6.376004849762261e-06 trainer/global_step‚11199$?±Í²)Ç & lr-AdamW/pg1‚6.376004849762261e-06 & lr-AdamW/pg2‚6.376004849762261e-06 trainer/global_step‚11199 % _timestamp‚1.7884470979921188e+09 _runtime‚9538.036799586 _step‚919—,$ÌÂ2³)- _wandbruntime‚9538  _runtime‚9538½²í×#(´)P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251503751302214 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1271574144526848 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185234334 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1069945252950016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374701527 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚9538.037865905 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1250781366264832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20795404862976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209093912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1250781366264832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20792804653568 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107493832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165627655 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1271576771127808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20795404862976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1271574144526848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165627655 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1098409814538240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20758316372038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77740035 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209093912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28464484960256 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1271576771127808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20792804653568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209073878 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107493835 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1230745460358144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1250781339873280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209073878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1250781339873280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20760914656706 % _timestamp‚1.7884470979929285e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165627649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0"‚µµÝ"µ)¢P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77740499 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1251506375014850 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185233867 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1098409737910272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165627649 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721567 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1271574144526848 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185234334 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1069945252950016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374701527 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚9538.037865905 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1250781366264832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20795404862976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209093912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1250781366264832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20792804653568 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107493832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165627655 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1271576771127808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20795404862976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1271574144526848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165627655 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1098409814538240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20758316372038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77740035 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209093912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28464484960256 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1271576771127808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20792804653568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209073878 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107493835 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1230745460358144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1250781339873280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209073878 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1250781339873280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20760914656706 % _timestamp‚1.7884470979929285e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165627649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251503751302214 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@Deû†KviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209092112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1250889305970176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374733997 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20760145714706 _step‚921 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77747155 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1098504728579072 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107503031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107503028 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1070037669137920 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1251611921819154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20762743999582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1251614545531998 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209112172 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 trainer/global_step‚11199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209092112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1271683916560896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20794636982272 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1098504651964416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1271686543175168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165641885 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185249734 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165641885 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28467010960896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚9538.783521474 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1230851801532416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1250889279578624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165641891 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165641891 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575/õM2'¸)¬N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165641891 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165641891 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209092112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1250889305970176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374733997 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20760145714706 _step‚921 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚77747155 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1098504728579072 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107503031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107503028 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1070037669137920 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1251611921819154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20762743999582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1251614545531998 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209112172 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 trainer/global_step‚11199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209092112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1271683916560896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20794636982272 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1098504651964416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1271686543175168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165641885 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185249734 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165641885 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28467010960896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 _runtime‚9538.783521474 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1230851801532416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1250889279578624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0™Uê¦2¹)- _wandbruntime‚9538  _runtime‚9538ß¾µf`º)Ú /loss/FRt3/assignment_loss‚2.936658561229706 /loss/FRt3/detection_loss‚2.4363852033002917 /loss/FBt1/detection_loss‚0.5420830002704451 2loss/SRqqt4/assignment_loss‚1.2330410084687173 0loss/FBt2/assignment_loss‚0.1406045426701894 /loss/FBt3/detection_loss‚2.8940577507019043 .loss/FBt4/detection_loss‚2.208387130498886 epoch‚9 1loss/FBt1/assignment_loss‚0.13468525525068553 2loss/FBt3/assignment_loss‚0.004701096273493022 0loss/FRt2/assignment_loss‚2.5656808064534116 2loss/SRqqt1/assignment_loss‚0.6482478856144811 1loss/FBt4/assignment_loss‚0.05451964265703282 1loss/SRqqt2/detection_loss‚1.5861144460878283 % _timestamp‚1.7884470987401307e+09 1loss/SRqqt1/detection_loss‚0.8057364512568534 _runtime‚9538.784733046 0loss/FRt4/assignment_loss‚2.2726643411886123 1loss/SRqqt4/detection_loss‚1.6641571188035111 /loss/FBt2/detection_loss‚2.1275007327397666 0loss/FRt1/assignment_loss‚0.8251890553984531 1loss/SRqqt3/assignment_loss‚1.375954047696931 .loss/FRt2/detection_loss‚2.468932047104224 /loss/FRt4/detection_loss‚2.3418465015434085 1loss/SRqqt3/detection_loss‚2.2442766794136593 %loss/total_loss‚36.51934697112716 2loss/SRqqt2/assignment_loss‚1.3599040840383407 /loss/FRt1/detection_loss‚1.6480195824663306 trainer/global_step‚11199 _step‚922û!xe»)ß 0loss/FRt1/assignment_loss‚0.8251890553984531 1loss/SRqqt3/assignment_loss‚1.375954047696931 .loss/FRt2/detection_loss‚2.468932047104224 /loss/FRt4/detection_loss‚2.3418465015434085 1loss/SRqqt3/detection_loss‚2.2442766794136593 %loss/total_loss‚36.51934697112716 2loss/SRqqt2/assignment_loss‚1.3599040840383407 /loss/FRt1/detection_loss‚1.6480195824663306 trainer/global_step‚11199 _step‚922 /loss/FRt3/assignment_loss‚2.936658561229706 /loss/FRt3/detection_loss‚2.4363852033002917 /loss/FBt1/detection_loss‚0.5420830002704451 2loss/SRqqt4/assignment_loss‚1.2330410084687173 0loss/FBt2/assignment_loss‚0.1406045426701894 /loss/FBt3/detection_loss‚2.8940577507019043 .loss/FBt4/detection_loss‚2.208387130498886 epoch‚9 1loss/FBt1/assignment_loss‚0.13468525525068553 2loss/FBt3/assignment_loss‚0.004701096273493022 0loss/FRt2/assignment_loss‚2.5656808064534116 2loss/SRqqt1/assignment_loss‚0.6482478856144811 1loss/FBt4/assignment_loss‚0.05451964265703282 1loss/SRqqt2/detection_loss‚1.5861144460878283 % _timestamp‚1.7884470987401307e+09 1loss/SRqqt1/detection_loss‚0.8057364512568534 _runtime‚9538.784733046 0loss/FRt4/assignment_loss‚2.2726643411886123 1loss/SRqqt4/detection_loss‚1.6641571188035111 /loss/FBt2/detection_loss‚2.1275007327397666šã†i’¼)‚(:‡ û’æÔØ›É gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K(sÔ2½)‚(:§ û’æÔä¼®Ë disk.sda2.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2685.375' disk.nvme0c0n1.out‚81832.7900390625 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.810546875 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.dm-1.out‚ 349.90625 disk.rbd3.out‚0 disk.sda3.out‚ 349.90625* proc.memory.percent‚16.390228271484375 disk.sda1.out‚0 cpu‚25.33366787691158% disk.nvme0n1.out‚81832.7900390625 network.sent‚ 2.8332722e+07 network.recv‚ 3.45298e+06# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-0.in‚ 1778.26171875" disk.dm-0.out‚81832.7900390625 disk.drbd1777.in‚0! disk.rbd2.out‚655.25341796875% disk./.usageGB‚1107.6899681091309 disk.dm-1.in‚0.5625 disk.sda.in‚0.5625$ memory_percent‚93.97993087768555 proc.cpu.threads‚17 disk.sda2.in‚0! disk.nvme0n1.in‚ 1778.26171875& proc.memory.availableMB‚ 986.328125) disk./.usagePercent‚74.33800317359228 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.out‚ 349.90625 disk.rbd3.in‚0ÃÅÛ”¾)‚(:‰ Š“æÔöµ÷È gpu.0.gpu‚70.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚245.977' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1954 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°³©ìC¿)‚(:¸ Š“æÔ€›ÆÌ disk.drbd1777.out‚0! disk.rbd2.out‚655.25341796875 cpu‚25.333217307038115 disk.sda1.in‚0& disk.nvme0n1.out‚81833.88916015625 disk.sda3.in‚0.5625 disk.dm-1.out‚ 350.23046875 proc.memory.rssMB‚2685.375 disk.rbd1.out‚ 4.8427734375 network.recv‚ 3.459101e+06 disk.sda2.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚74.33794966801763 disk.sda.in‚0.5625 disk.sda2.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.rbd0.out‚0 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.in‚0$ memory_percent‚93.97885799407959 network.sent‚ 2.8441146e+07# disk.dm-0.out‚81833.88916015625( proc.memory.availableMB‚ 986.50390625* proc.memory.percent‚16.390228271484375 disk.sda.out‚ 350.23046875 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.rbd1.in‚0 disk.sda3.out‚ 350.23046875( disk.nvme0c0n1.out‚81833.88916015625 disk.rbd2.in‚0% disk./.usageGB‚1107.6891708374023 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5625âE¢l”À)‚(:‰ ™“æÔýå›É gpu.0.gpu‚72.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚261.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.2772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f'ÅŒ@Á)‚(:µ ™“æÔàïË disk.sda3.in‚0.5625 disk.drbd1777.out‚0 disk.sda.in‚0.5625( disk.nvme0c0n1.out‚81834.20556640625 proc.memory.rssMB‚2685.375 disk.drbd1777.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.dm-0.in‚ 1778.26171875* proc.memory.percent‚16.390228271484375 cpu‚25.33278107294152# disk.dm-0.out‚81834.20556640625 network.recv‚ 3.463266e+06! disk.rbd2.out‚656.90576171875 disk.rbd3.in‚0 disk.dm-1.out‚ 350.62109375# disk.nvme0c0n1.in‚ 1778.26171875 disk.sda.out‚ 350.62109375 disk.sda1.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 986.265625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda3.out‚ 350.62109375% disk./.usageGB‚1107.8146286010742 disk.rbd1.out‚ 4.8427734375& disk.nvme0n1.out‚81834.20556640625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚93.98031234741211 network.sent‚ 2.8446981e+07) disk./.usagePercent‚74.34636924380371 disk.sda2.in‚0 disk.rbd3.out‚0Á¾üª2Â)- _wandbruntime‚9577  _runtime‚9577¬©ÌÈÃ) & lr-AdamW/pg1‚5.683904353989622e-06 & lr-AdamW/pg2‚5.683904353989622e-06 trainer/global_step‚11249 % _timestamp‚1.7884471372973983e+09 _runtime‚9577.342071293 _step‚923†r‹ÍÄ)Ç _runtime‚9577.342071293 _step‚923 & lr-AdamW/pg1‚5.683904353989622e-06 & lr-AdamW/pg2‚5.683904353989622e-06 trainer/global_step‚11249 % _timestamp‚1.7884471372973983e+09›:–°2Å)- _wandbruntime‚9577  _runtime‚9577€u3NG Æ)P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20887035712000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1256178325140480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 trainer/global_step‚11249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107953632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1256178325140480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209988930 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28590004772352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1277062760643072 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1074564655625216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20884435502592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78075786 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884471372981696e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 trainer/global_step‚11249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107953632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1256178325140480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209988930 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28590004772352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1277062760643072 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1074564655625216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20884435502592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78075786 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884471372981696e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753œæú"2È)- _wandbruntime‚9578  _runtime‚9578Æ\õP-'É)§N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210007164 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210007164 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107962831 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78082894 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186045278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1236168860246016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20854224724190 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 trainer/global_step‚11249 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1257023084970206 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20851626439314 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166353685 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1256286264845824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166353691 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107962828 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28592505321472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 _runtime‚9578.086719957 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20886267831296 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186045725 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1074657043678720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1277172532677120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166353691 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20888868054016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1103249549000192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1074657071813120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210027224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1256286291237376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20888868054016 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1257020461257362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1256286291237376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210027224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1256286264845824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1277175159291392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1277175159291392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360849 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1236168834818048EO£zÊ)¬N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1257020461257362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1256286291237376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210027224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1256286264845824 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1277175159291392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1277175159291392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376360849 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1236168834818048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210007164 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210007164 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107962831 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78082894 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186045278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1236168860246016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20854224724190 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 trainer/global_step‚11249 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1257023084970206 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20851626439314 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166353685 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1256286264845824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166353691 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107962828 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28592505321472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 _runtime‚9578.086719957 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20886267831296 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186045725 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1074657043678720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1277172532677120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166353691 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20888868054016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1103249549000192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1074657071813120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210027224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1256286291237376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20888868054016 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575|‘ch2Ë)- _wandbruntime‚9578  _runtime‚9578”*D^Ì)Ø epoch‚9 0loss/FRt2/assignment_loss‚2.4279441752217035 &loss/total_loss‚36.769231176226896 % _timestamp‚1.7884471380431626e+09 .loss/FRt3/detection_loss‚2.432895139131038 1loss/SRqqt1/detection_loss‚0.8467541943265967 1loss/SRqqt3/detection_loss‚2.2689613589218687 0loss/FRt4/assignment_loss‚2.0384887993335723 1loss/FBt3/assignment_loss‚0.18519035360077396 2loss/SRqqt1/assignment_loss‚0.7278974455947133 2loss/SRqqt3/assignment_loss‚1.7671732442719597 1loss/FBt2/assignment_loss‚0.08089896257400905 1loss/SRqqt2/detection_loss‚1.6001356288529278 .loss/FBt4/detection_loss‚2.916542038321495 .loss/FRt1/detection_loss‚1.617852982878685 2loss/SRqqt2/assignment_loss‚1.4459453231344621 1loss/FBt1/assignment_loss‚0.05505764620962071 /loss/FBt1/detection_loss‚0.6341692398968412 2loss/FBt4/assignment_loss‚0.005538146069739014 -loss/FRt2/detection_loss‚2.33816031772982 1loss/SRqqt4/detection_loss‚1.6443180465698242 /loss/FBt2/detection_loss‚2.0716886264937266 .loss/FBt3/detection_loss‚2.368055447936058 trainer/global_step‚11249 _runtime‚9578.087750119 _step‚926 0loss/FRt1/assignment_loss‚0.9426228976249695 0loss/FRt3/assignment_loss‚2.9932793276650567 2loss/SRqqt4/assignment_loss‚1.1653903759270907 -loss/FRt4/detection_loss‚2.19427145794034" /ºcÍ)Ý 1loss/FBt2/assignment_loss‚0.08089896257400905 1loss/SRqqt2/detection_loss‚1.6001356288529278 .loss/FBt4/detection_loss‚2.916542038321495 .loss/FRt1/detection_loss‚1.617852982878685 2loss/SRqqt2/assignment_loss‚1.4459453231344621 1loss/FBt1/assignment_loss‚0.05505764620962071 /loss/FBt1/detection_loss‚0.6341692398968412 2loss/FBt4/assignment_loss‚0.005538146069739014 -loss/FRt2/detection_loss‚2.33816031772982 1loss/SRqqt4/detection_loss‚1.6443180465698242 /loss/FBt2/detection_loss‚2.0716886264937266 .loss/FBt3/detection_loss‚2.368055447936058 trainer/global_step‚11249 _runtime‚9578.087750119 _step‚926 0loss/FRt1/assignment_loss‚0.9426228976249695 0loss/FRt3/assignment_loss‚2.9932793276650567 2loss/SRqqt4/assignment_loss‚1.1653903759270907 -loss/FRt4/detection_loss‚2.19427145794034 epoch‚9 0loss/FRt2/assignment_loss‚2.4279441752217035 &loss/total_loss‚36.769231176226896 % _timestamp‚1.7884471380431626e+09 .loss/FRt3/detection_loss‚2.432895139131038 1loss/SRqqt1/detection_loss‚0.8467541943265967 1loss/SRqqt3/detection_loss‚2.2689613589218687 0loss/FRt4/assignment_loss‚2.0384887993335723 1loss/FBt3/assignment_loss‚0.18519035360077396 2loss/SRqqt1/assignment_loss‚0.7278974455947133 2loss/SRqqt3/assignment_loss‚1.7671732442719597žµR¡”Î)‚(:‰ ¨“æÔúŠ¥É gpu.0.gpu‚69.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚227.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.4136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C#Ú5Ï)‚(:ª ¨“æÔùƒÀË! disk.nvme0n1.in‚ 1778.26171875& disk.nvme0n1.out‚81931.91650390625 disk.rbd0.out‚0 disk.sda.out‚351.1875 network.sent‚ 2.8452986e+07 network.recv‚ 3.467846e+06 cpu‚25.33231304795404# disk.nvme0c0n1.in‚ 1778.26171875* proc.memory.percent‚16.390228271484375! disk.rbd2.out‚656.90576171875 disk.rbd3.out‚0# disk.dm-0.out‚81931.91650390625 disk.dm-1.out‚351.1875 disk.sda.in‚0.5625 disk.rbd3.in‚0& proc.memory.availableMB‚ 986.171875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1107.8141708374023 disk.drbd1777.out‚0 disk.rbd2.in‚0$ memory_percent‚93.98088455200195) disk./.usagePercent‚74.34633852289961 disk.sda2.in‚0 disk.sda3.in‚0.5625 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.90576171875 disk.sda3.out‚351.1875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚81931.91650390625 disk.dm-0.in‚ 1778.26171875 disk.dm-1.in‚0.5625 proc.memory.rssMB‚2685.375Ùp!ø’Ð)‚(:‡ ·“æÔð͈É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ–*-;Ñ)‚(:° ·“æÔþÛ÷Ê disk.sda3.out‚ 351.78125 disk.sda.in‚0.5625 disk.sda2.out‚0 disk.dm-0.in‚ 1778.26171875 disk.rbd2.in‚0! disk.rbd2.out‚657.04638671875 disk.dm-1.out‚ 351.78125 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.90576171875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚74.36318074656221 disk.sda.out‚ 351.78125 disk.rbd3.in‚0( proc.memory.availableMB‚ 986.16796875% disk./.usageGB‚1108.0651321411133 disk.sda3.in‚0.5625# disk.nvme0c0n1.in‚ 1778.26171875( disk.nvme0c0n1.out‚81932.22900390625 disk.sda2.in‚0 cpu‚25.331876297049114# disk.dm-0.out‚81932.22900390625 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1778.26171875$ memory_percent‚93.98090839385986 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚81932.22900390625 proc.memory.rssMB‚2685.375* proc.memory.percent‚16.390228271484375 network.sent‚ 2.8562309e+07 network.recv‚ 3.474481e+062?§®”Ò)‚(:‰ Æ“æÔðŠ¬É gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚257.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ѭ ”7Ó)‚(:¬ Æ“æÔŠŽìË! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.out‚ 352.28515625 disk.drbd1777.in‚0 disk.sda.out‚ 352.28515625( proc.memory.availableMB‚ 986.38671875 cpu‚25.331432785684534 network.recv‚ 3.478047e+06) disk./.usagePercent‚74.35141208021318& disk.nvme0c0n1.out‚82030.521484375 disk.dm-0.in‚ 1778.26171875 network.sent‚ 2.8567615e+07$ disk.nvme0n1.out‚82030.521484375 disk.sda1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0.5625 disk.rbd0.in‚0 disk.dm-1.out‚ 352.28515625 disk.rbd2.in‚0 disk.sda.in‚0.5625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0# memory_percent‚93.9795732498169* proc.memory.percent‚16.390228271484375 disk.sda2.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1778.26171875! disk.dm-0.out‚82030.521484375 disk.dm-1.in‚0.5625! disk.rbd2.out‚657.04638671875 proc.memory.rssMB‚2685.375% disk./.usageGB‚1107.8897705078125 disk.drbd1777.out‚0 disk.rbd1.out‚4.96875Ý6H.2Ô)- _wandbruntime‚9616  _runtime‚9616Ø*.ÈÕ) _step‚927 & lr-AdamW/pg1‚5.030998019259181e-06 & lr-AdamW/pg2‚5.030998019259181e-06 trainer/global_step‚11299 % _timestamp‚1.7884471765336251e+09 _runtime‚9616.578268913*tÙÞÍÖ)Ç _runtime‚9616.578268913 _step‚927 & lr-AdamW/pg1‚5.030998019259181e-06 & lr-AdamW/pg2‚5.030998019259181e-06 trainer/global_step‚11299 % _timestamp‚1.7884471765336251e+09Ÿ©AÑd2×)- _wandbruntime‚9616  _runtime‚9616TÓr#(Ø)P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1282554003360256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1282554003360256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1079184086434816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20943876105922 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20978666561024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1241379552357376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 % _timestamp‚1.7884471765345027e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20976066351616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186824775 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78411343 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167051255 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1261575336799232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1261575310407680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167051249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108413432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20976066351616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1282551376759296 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1282551376759296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210924016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78411780 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28715813983232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1261575310407680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210903982 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20978666561024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 _step‚928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167051255 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1261575336799232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210903982 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1079184058300416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210924016 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1107899900418048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28715767586816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 _runtime‚9616.579491136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920ÂCøYh'Ù)¢P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78411343 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167051255 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1261575336799232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1261575310407680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167051249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108413432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20976066351616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1282551376759296 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1282551376759296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210924016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78411780 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28715813983232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1261575310407680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210903982 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20978666561024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 _step‚928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167051255 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1261575336799232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210903982 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1079184058300416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210924016 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1107899900418048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28715767586816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 _runtime‚9616.579491136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1282554003360256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1282554003360256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1079184086434816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20943876105922 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20978666561024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1241379552357376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 % _timestamp‚1.7884471765345027e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20976066351616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactives²G6À_split.all.freed‚ 186824775 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 û¾ý2Ú)- _wandbruntime‚9617  _runtime‚9617ØBiÅ-'Û)§N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78418198 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1107994815485952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1262429000695570 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚929 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210942276 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20945705448798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261683276504576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28718340997632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1241485918959616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1241485893531648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1261683250113024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186841274 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108422631 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78418643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20943107163922 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1282663775407616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210922216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 _runtime‚9617.545633603 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167065491 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1261683250113024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1282661148793344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20977898680320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186840826 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20980498903040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261683276504576 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108422628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28718292517376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20977898680320 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167065485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1282663775407616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167065491 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167065485 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20945705448798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261683276504576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚28718340997632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1241485918959616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1241485893531648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1261683250113024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186841274 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108422631 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78418643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20943107163922 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1282663775407616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210922216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 _runtime‚9617.545633603 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167065491 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1261683250113024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1282661148793344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20977898680320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186840826 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20980498903040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261683276504576 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108422628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28718292517376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20977898680320 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167065485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1282663775407616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167065491 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167065485 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚929¡—2Ý)- _wandbruntime‚9617  _runtime‚9617r6µ›^Þ)Ø 2loss/SRqqt2/assignment_loss‚0.9234558006003499 _runtime‚9617.546696465 1loss/FBt4/assignment_loss‚0.16529649997751222 epoch‚9 0loss/FRt1/assignment_loss‚1.0239990314711696 1loss/FBt1/assignment_loss‚0.03758534216729286 .loss/FRt1/detection_loss‚1.739024605035134 /loss/FRt2/detection_loss‚2.3941453523933887 1loss/SRqqt1/detection_loss‚0.8848050058339582 2loss/SRqqt1/assignment_loss‚0.7409581339205474 2loss/SRqqt3/assignment_loss‚1.4342643133340738 1loss/SRqqt4/detection_loss‚1.7648189928043971 0loss/FBt1/detection_loss‚0.46194790736560454 /loss/FBt3/detection_loss‚2.2506018757820128 /loss/FBt4/detection_loss‚1.7286833822727203 /loss/FRt2/assignment_loss‚2.010838030278683 1loss/FBt3/assignment_loss‚0.14223189293406904 &loss/total_loss‚34.914041739629766 0loss/FRt3/assignment_loss‚2.6781865590956153 /loss/FRt4/assignment_loss‚2.441087821696667 /loss/SRqqt3/detection_loss‚2.03021915218769 % _timestamp‚1.7884471775021257e+09 1loss/FBt2/assignment_loss‚0.24146923879161478 0loss/SRqqt2/detection_loss‚1.694592393040657 .loss/FBt2/detection_loss‚2.348710227012634 _step‚930 2loss/SRqqt4/assignment_loss‚1.0134374724870379 /loss/FRt3/detection_loss‚2.3596015465268354 .loss/FRt4/detection_loss‚2.404081162620098 trainer/global_step‚11299Zþ‚cß)Ý 0loss/FRt3/assignment_loss‚2.6781865590956153 /loss/FRt4/assignment_loss‚2.441087821696667 /loss/SRqqt3/detection_loss‚2.03021915218769 % _timestamp‚1.7884471775021257e+09 1loss/FBt2/assignment_loss‚0.24146923879161478 0loss/SRqqt2/detection_loss‚1.694592393040657 .loss/FBt2/detection_loss‚2.348710227012634 _step‚930 2loss/SRqqt4/assignment_loss‚1.0134374724870379 /loss/FRt3/detection_loss‚2.3596015465268354 .loss/FRt4/detection_loss‚2.404081162620098 trainer/global_step‚11299 2loss/SRqqt2/assignment_loss‚0.9234558006003499 _runtime‚9617.546696465 1loss/FBt4/assignment_loss‚0.16529649997751222 epoch‚9 0loss/FRt1/assignment_loss‚1.0239990314711696 1loss/FBt1/assignment_loss‚0.03758534216729286 .loss/FRt1/detection_loss‚1.739024605035134 /loss/FRt2/detection_loss‚2.3941453523933887 1loss/SRqqt1/detection_loss‚0.8848050058339582 2loss/SRqqt1/assignment_loss‚0.7409581339205474 2loss/SRqqt3/assignment_loss‚1.4342643133340738 1loss/SRqqt4/detection_loss‚1.7648189928043971 0loss/FBt1/detection_loss‚0.46194790736560454 /loss/FBt3/detection_loss‚2.2506018757820128 /loss/FBt4/detection_loss‚1.7286833822727203 /loss/FRt2/assignment_loss‚2.010838030278683 1loss/FBt3/assignment_loss‚0.14223189293406904 &loss/total_loss‚34.914041739629766¢Ž:W”à)‚(:‰ Õ“æÔÛÑžÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚277.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w‚äY3á)‚(:¨ Õ“æÔ¸Ù¢Ë disk.dm-1.out‚ 353.02734375 disk.drbd1777.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 986.33984375 cpu‚25.330998747940058 network.recv‚ 3.48249e+06) disk./.usagePercent‚74.35140824010017 disk.rbd3.out‚0 disk.sda3.out‚ 353.02734375& disk.nvme0c0n1.out‚82030.841796875 proc.memory.rssMB‚2685.375% disk./.usageGB‚1107.8897132873535 disk.sda.in‚0.5625 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.rbd3.in‚0$ disk.nvme0n1.out‚82030.841796875 disk.sda3.in‚0.5625 disk.sda2.in‚0! disk.dm-0.out‚82030.841796875 disk.sda.out‚ 353.02734375 disk.rbd1.out‚4.96875 disk.rbd2.out‚ 657.2109375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1778.26171875$ memory_percent‚93.97985935211182* proc.memory.percent‚16.390228271484375 disk.rbd0.out‚0 network.sent‚ 2.8573436e+07 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1778.26171875|Ÿ”â)‚(:‰ ä“æÔâÄÁÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.426' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úv÷©7ã)‚(:¬ ä“æÔæÏË disk.rbd1.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.rbd3.out‚0 disk.dm-1.out‚ 353.4765625$ memory_percent‚93.98167133331299( proc.memory.availableMB‚ 986.04296875% disk./.usageGB‚1107.8897705078125( disk.nvme0c0n1.out‚82128.29833984375 disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚82128.29833984375 disk.sda.in‚0.5625 disk.sda2.out‚0 cpu‚25.330561052727937 disk.sda2.in‚0 network.recv‚ 3.488967e+06 disk.rbd1.out‚4.96875 disk.rbd2.out‚ 657.2109375 disk.rbd3.in‚0 proc.memory.rssMB‚2685.375 disk.sda1.in‚0 disk.sda3.in‚0.5625 disk.drbd1777.out‚0* proc.memory.percent‚16.390228271484375 disk.sda1.out‚0 network.sent‚ 2.8682473e+07 disk.rbd2.in‚0 disk.sda3.out‚ 353.4765625 disk.sda.out‚ 353.4765625# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚0.5625 disk.drbd1777.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚82128.29833984375 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚74.35141208021318Ô‰Ì2ä)- _wandbruntime‚9655  _runtime‚9655æ¯å*Èå) & lr-AdamW/pg1‚4.417421910272524e-06 & lr-AdamW/pg2‚4.417421910272524e-06 trainer/global_step‚11349 % _timestamp‚1.7884472159318066e+09 _runtime‚9655.976483856 _step‚931™ä"›Íæ)Ç & lr-AdamW/pg2‚4.417421910272524e-06 trainer/global_step‚11349 % _timestamp‚1.7884472159318066e+09 _runtime‚9655.976483856 _step‚931 & lr-AdamW/pg1‚4.417421910272524e-06£ þUV2ç)- _wandbruntime‚9655  _runtime‚9655™ø3›Áè)œP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 $ _timestamp‚1.788447215932638e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211839068 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167763049 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1267731993329474 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1288042619476480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1246696611070976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211839068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379602123 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚9655.977443725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21067697200640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167763055 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚482 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1112645174896640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _step‚932 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1288039992875520 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1288039992875520 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187620190 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1266972295674880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167763049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167763055 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083803460975616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1246696636498944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1267729369616838 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 J‚S|naBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚485 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21035356830530 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167763055 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1083803460975616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1246696636498944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1267729369616838 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚485 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21035356830530 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 $ _timestamp‚1.788447215932638e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211839068 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167763049 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1267731993329474 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1288042619476480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1246696611070976 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211839068 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379602123 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚9655.977443725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21067697200640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167763055 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚482 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1112645174896640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 _step‚932 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1288039992875520 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1288039992875520¤R¹2ê)- _wandbruntime‚9656  _runtime‚9656h ¸-'ë)§N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 trainer/global_step‚11349 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21069529529344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1083895849029120 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28844215322112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211837268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187636045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1267080261771776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21072129752064 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1267080261771776 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1246802977673216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1267080235380224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1288149764909568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1246802952245248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1112740064351232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083895877163520 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1267840163846622 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187636510 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884472166782892e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634619 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21072129752064 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1267080261771776 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1246802977673216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1267080235380224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1288149764909568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1246802952245248 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1112740064351232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1083895877163520 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1267840163846622 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187636510 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 % _timestamp‚1.7884472166782892e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634619 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚28844215322112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211837268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187636045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitoÿ¨’4<r.on_train_batch_end/active_bytes.all.peak‚ 20788341248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1267080261771776¥©×Ó2í)- _wandbruntime‚9656  _runtime‚9656°$ˆ÷bî)Ü 2loss/SRqqt1/assignment_loss‚0.8934819754037373 1loss/FBt4/assignment_loss‚0.07710934577465361 %loss/total_loss‚36.21848064317257 _step‚934 0loss/FRt1/assignment_loss‚0.9923100288097675 .loss/FRt4/detection_loss‚2.326545049182393 1loss/SRqqt1/detection_loss‚1.0829810158542867 0loss/FRt3/assignment_loss‚2.9089081081850776 2loss/FBt1/assignment_loss‚0.039119797468081134 1loss/SRqqt3/detection_loss‚2.3396186559431014 .loss/FBt3/detection_loss‚2.289339522520701 .loss/FBt4/detection_loss‚2.045631143179807 _runtime‚9656.724561244 0loss/FRt4/assignment_loss‚2.1518695936961607 2loss/FBt3/assignment_loss‚0.007803813394275494 /loss/FRt2/detection_loss‚2.2200779454575645 0loss/SRqqt4/detection_loss‚1.625729369206561 /loss/FBt2/detection_loss‚2.3662099316716194 2loss/SRqqt3/assignment_loss‚1.6659195812479142 2loss/FBt2/assignment_loss‚0.015463784358871635 0loss/SRqqt2/detection_loss‚1.935373840435427 epoch‚9 /loss/FBt1/detection_loss‚0.6297794973943383 2loss/SRqqt2/assignment_loss‚1.1115363417953557 2loss/SRqqt4/assignment_loss‚1.1666413994368028 /loss/FRt1/detection_loss‚1.3755454696858158 /loss/FRt3/detection_loss‚2.8354258330878896 trainer/global_step‚11349 % _timestamp‚1.7884472166799603e+09 0loss/FRt2/assignment_loss‚2.1160595999823677 ±OZgï)á 2loss/SRqqt3/assignment_loss‚1.6659195812479142 2loss/FBt2/assignment_loss‚0.015463784358871635 0loss/SRqqt2/detection_loss‚1.935373840435427 epoch‚9 /loss/FBt1/detection_loss‚0.6297794973943383 2loss/SRqqt2/assignment_loss‚1.1115363417953557 2loss/SRqqt4/assignment_loss‚1.1666413994368028 /loss/FRt1/detection_loss‚1.3755454696858158 /loss/FRt3/detection_loss‚2.8354258330878896 trainer/global_step‚11349 % _timestamp‚1.7884472166799603e+09 0loss/FRt2/assignment_loss‚2.1160595999823677 2loss/SRqqt1/assignment_loss‚0.8934819754037373 1loss/FBt4/assignment_loss‚0.07710934577465361 %loss/total_loss‚36.21848064317257 _step‚934 0loss/FRt1/assignment_loss‚0.9923100288097675 .loss/FRt4/detection_loss‚2.326545049182393 1loss/SRqqt1/detection_loss‚1.0829810158542867 0loss/FRt3/assignment_loss‚2.9089081081850776 2loss/FBt1/assignment_loss‚0.039119797468081134 1loss/SRqqt3/detection_loss‚2.3396186559431014 .loss/FBt3/detection_loss‚2.289339522520701 .loss/FBt4/detection_loss‚2.045631143179807 _runtime‚9656.724561244 0loss/FRt4/assignment_loss‚2.1518695936961607 2loss/FBt3/assignment_loss‚0.007803813394275494 /loss/FRt2/detection_loss‚2.2200779454575645 0loss/SRqqt4/detection_loss‚1.625729369206561 /loss/FBt2/detection_loss‚2.3662099316716194¦ùÄ ð)‚(:’ ó“æÔüÿšÉ gpu.0.gpu‚78.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚270.34' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.06799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Á¬)ñ)‚(:ž ó“æÔŒ¿—Ì) disk./.usagePercent‚74.35140849610771 disk.rbd1.in‚0 disk.rbd1.out‚4.96875 disk.sda1.in‚0 disk.sda2.out‚0% disk./.usageGB‚1107.8897171020508 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd2.out‚ 658.92578125 disk.sda1.out‚0% disk.nvme0n1.out‚82128.6318359375 proc.memory.rssMB‚2685.375 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚16.390228271484375 disk.sda.in‚0.5625 disk.sda2.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.out‚354.0625$ memory_percent‚93.98326873779297% proc.memory.availableMB‚ 985.78125# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.out‚0 disk.sda.out‚354.0625 disk.rbd3.out‚0 disk.sda3.in‚0.5625 proc.cpu.threads‚17 network.sent‚ 2.8688694e+07 disk.rbd0.out‚0' disk.nvme0c0n1.out‚82128.6318359375 cpu‚25.330123589079435" disk.dm-0.out‚82128.6318359375 disk.dm-1.out‚354.0625 network.recv‚ 3.493537e+06ºÓëO”ò)‚(:‰ ‚”æÔËß²É gpu.0.gpu‚62.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚47.0 gpu.0.powerWatts‚139.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚099 ¿3ó)‚(:¨ ‚”æÔÙ…·Ë disk.sda2.in‚0# disk.dm-0.out‚82226.42333984375 disk.drbd1777.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚82226.42333984375 disk.sda3.in‚0.5625 disk.sda3.out‚ 354.5234375 disk.rbd0.out‚0 cpu‚25.3296687934312 network.recv‚ 3.499373e+06 disk.dm-1.out‚ 354.5234375 disk.rbd2.out‚ 658.92578125 network.sent‚ 2.8797135e+07$ disk./.usageGB‚1107.864315032959 disk.rbd3.out‚0 disk.rbd1.out‚4.96875 disk.sda.in‚0.5625 proc.cpu.threads‚17 disk.rbd2.in‚0( disk./.usagePercent‚74.3497037419372 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0.5625 disk.rbd1.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚82226.42333984375 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.out‚0 disk.sda.out‚ 354.5234375' proc.memory.availableMB‚ 986.2890625 proc.memory.rssMB‚2685.375# disk.nvme0c0n1.in‚ 1778.26171875$ memory_percent‚93.98016929626465* proc.memory.percent‚16.390228271484375³"¤ô)‚(:’ ‘”æÔ‚ãÉ gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.34' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.06799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×N4õ)‚(:© ‘”æÔάË network.sent‚ 2.8803013e+07 disk.dm-1.in‚0.5625 disk.rbd0.in‚0 disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚82226.72412109375 disk.sda.in‚0.5625 disk.sda.out‚ 354.90625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚82226.72412109375 cpu‚25.32923933877634 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0 network.recv‚ 3.503151e+06 disk.rbd1.in‚0 disk.rbd1.out‚4.96875 disk.sda3.in‚0.5625$ memory_percent‚93.97919178009033) disk./.usagePercent‚74.34970246189953 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1778.26171875% disk./.usageGB‚1107.8642959594727! disk.rbd2.out‚659.02392578125 disk.sda3.out‚ 354.90625 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1778.26171875( disk.nvme0c0n1.out‚82226.72412109375* proc.memory.percent‚16.390228271484375 proc.cpu.threads‚17 disk.dm-1.out‚ 354.90625 disk.sda1.in‚0( proc.memory.availableMB‚ 986.44921875 proc.memory.rssMB‚2685.375O Ú2ö)- _wandbruntime‚9695  _runtime‚96957â@lÆ÷)À trainer/global_step‚11399 % _timestamp‚1.7884472554043188e+09 _runtime‚9695.448989509 _step‚935 % lr-AdamW/pg1‚3.84330389537107e-06 % lr-AdamW/pg2‚3.84330389537107e-06N°žñËø)Å trainer/global_step‚11399 % _timestamp‚1.7884472554043188e+09 _runtime‚9695.448989509 _step‚935 % lr-AdamW/pg1‚3.84330389537107e-06 % lr-AdamW/pg2‚3.84330389537107e-06§êãa2ù)- _wandbruntime‚9695  _runtime‚9695XË©ã"(ú)œP S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1293528608991744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21159328049664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28967290365952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚9695.457416212 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21161928259072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381228975 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168474849 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28967241872384 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79083591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168474849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21161928259072 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212734086 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212754120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21159328049664 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1117390182151168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11399 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1293531235592704 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212734086 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109333035 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79083133 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1273140532767682 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1272369280942080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109333032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188416626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188416165 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1252013695212544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168474855 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1293531235592704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21126837555138 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1252013669784576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1293528608991744Ü2”ï'(û)¡P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381228975 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168474849 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚28967241872384 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79083591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168474849 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21161928259072 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212734086 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212754120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21159328049664 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1117390182151168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11399 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1293531235592704 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212734086 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109333035 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79083133 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1273140532767682 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1272369280942080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109333032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188416626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188416165 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1252013695212544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168474855 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1293531235592704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21126837555138 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1252013669784576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1293528608991744 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1293528608991744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21159328049664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚28967290365952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚9695.457416212 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21161928259072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0¨fŠˆ2ü)- _wandbruntime‚9696  _runtime‚9696sÝ”†} ý)§N I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 _step‚937 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1088515251704320 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212752320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168489085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1293638381025792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79089778 _runtime‚9696.196298294 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21163760601088 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109342231 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212772380 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21163760601088 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1273246079571986 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1117485094878720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1252120036386816 trainer/global_step‚11399 % _timestamp‚1.7884472561511414e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1088515279838720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21126068613138 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1272477220647424 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212752320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241405 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188432006 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212772380 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21163760601088 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1273246079571986 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1117485094878720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1252120036386816 trainer/global_step‚11399 % _timestamp‚1.7884472561511414e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1088515279838720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21126068613138 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1272477220647424 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212752320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241405 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188432006 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1088515251704320 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212752320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168489085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1293638381025792 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79089778 _runtime‚9696.196298294 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21163760601088 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109342231 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074©rÂ2ÿ)- _wandbruntime‚9696  _runtime‚9696N¦`€*Ú trainer/global_step‚11399 1loss/FBt1/assignment_loss‚0.12318341008573595 -loss/FBt2/detection_loss‚2.78760302066803 0loss/SRqqt4/detection_loss‚1.701636419992666 /loss/FRt1/detection_loss‚1.6713576913267996 0loss/FRt4/assignment_loss‚2.4441909777621427 2loss/SRqqt4/assignment_loss‚1.1059411692300014 .loss/FRt4/detection_loss‚2.429323776314656 /loss/FRt3/detection_loss‚2.5359863435534695 1loss/SRqqt1/detection_loss‚1.0531505989176886 1loss/SRqqt3/detection_loss‚2.3632591168085733 /loss/FBt1/detection_loss‚0.5194534755481238 epoch‚9 0loss/FRt1/assignment_loss‚1.2059568762779236 /loss/FRt3/assignment_loss‚3.407220970500599 1loss/FBt2/assignment_loss‚0.10989066660404205 1loss/FBt4/assignment_loss‚0.12724450684618205 -loss/FBt4/detection_loss‚2.53487462302049 &loss/total_loss‚38.628220923517794 2loss/SRqqt2/assignment_loss‚0.9769929905111591 2loss/SRqqt3/assignment_loss‚1.4774074300728512 3loss/FBt3/assignment_loss‚0.0042886059381999075 1loss/SRqqt2/detection_loss‚1.7544610564615213 _runtime‚9696.197854817 2loss/SRqqt1/assignment_loss‚1.0119535014232888 /loss/FBt3/detection_loss‚2.5648195862770082 % _timestamp‚1.7884472561532521e+09 _step‚938 0loss/FRt2/assignment_loss‚2.2479203110787926 /loss/FRt2/detection_loss‚2.4701037982978473Nþ…­e*ß /loss/FRt1/detection_loss‚1.6713576913267996 0loss/FRt4/assignment_loss‚2.4441909777621427 2loss/SRqqt4/assignment_loss‚1.1059411692300014 .loss/FRt4/detection_loss‚2.429323776314656 /loss/FRt3/detection_loss‚2.5359863435534695 1loss/SRqqt1/detection_loss‚1.0531505989176886 1loss/SRqqt3/detection_loss‚2.3632591168085733 /loss/FBt1/detection_loss‚0.5194534755481238 epoch‚9 0loss/FRt1/assignment_loss‚1.2059568762779236 /loss/FRt3/assignment_loss‚3.407220970500599 1loss/FBt2/assignment_loss‚0.10989066660404205 1loss/FBt4/assignment_loss‚0.12724450684618205 -loss/FBt4/detection_loss‚2.53487462302049 &loss/total_loss‚38.628220923517794 2loss/SRqqt2/assignment_loss‚0.9769929905111591 2loss/SRqqt3/assignment_loss‚1.4774074300728512 3loss/FBt3/assignment_loss‚0.0042886059381999075 1loss/SRqqt2/detection_loss‚1.7544610564615213 _runtime‚9696.197854817 2loss/SRqqt1/assignment_loss‚1.0119535014232888 /loss/FBt3/detection_loss‚2.5648195862770082 % _timestamp‚1.7884472561532521e+09 _step‚938 0loss/FRt2/assignment_loss‚2.2479203110787926 /loss/FRt2/detection_loss‚2.4701037982978473 trainer/global_step‚11399 1loss/FBt1/assignment_loss‚0.12318341008573595 -loss/FBt2/detection_loss‚2.78760302066803 0loss/SRqqt4/detection_loss‚1.701636419992666ª„%ìež‚*‚(:“  ”æÔ¢ë’É gpu.0.gpu‚90.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚46.0 gpu.0.powerWatts‚262.448' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.48959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼–SBƒ*‚(:·  ”æÔËÃË disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚82324.42138671875) disk./.usagePercent‚74.36653009313268 disk.sda2.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2624.3359375 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 355.25390625 disk.dm-1.in‚0.5625 disk.dm-1.out‚ 355.25390625$ memory_percent‚93.62781047821045 disk.sda.out‚ 355.25390625& disk.nvme0n1.out‚82324.42138671875 disk.sda3.in‚0.5625* proc.memory.percent‚16.017675399780273 cpu‚25.32880614554487 disk.sda1.in‚0) proc.memory.availableMB‚ 1044.01953125 disk.drbd1777.in‚0 disk.sda.in‚0.5625! disk.nvme0n1.in‚ 1778.26171875( disk.nvme0c0n1.out‚82324.42138671875 network.sent‚ 2.8808828e+07 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 network.recv‚ 3.507507e+06% disk./.usageGB‚1108.1150398254395! disk.rbd2.out‚659.02392578125 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚4.96875 disk.rbd0.out‚0M¡L”„*‚(:‰ ¯”æÔ‡É gpu.0.gpu‚56.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚48.0 gpu.0.powerWatts‚264.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03²Áf7…*‚(:¬ ¯”æÔ©åóË network.sent‚ 2.891819e+07% disk.nvme0n1.out‚82324.9619140625 disk.rbd0.in‚0" disk.dm-0.out‚82324.9619140625! disk.nvme0n1.in‚ 1778.26171875 disk.drbd1777.out‚0 disk.rbd1.out‚4.96875 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚93.64452362060547# proc.memory.rssMB‚ 2626.97265625 disk.sda3.in‚0.5625& proc.memory.availableMB‚ 1041.28125! disk.rbd2.out‚659.16455078125$ disk./.usageGB‚1108.115104675293# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚0.5625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1778.26171875 disk.dm-1.out‚ 355.9609375 disk.rbd1.in‚0 disk.sda.in‚0.5625 disk.sda1.out‚0) proc.memory.percent‚16.03376865386963 cpu‚25.32834939818833 disk.rbd0.out‚0 network.recv‚ 3.514184e+06 disk.sda3.out‚ 355.9609375' disk.nvme0c0n1.out‚82324.9619140625 disk.drbd1777.in‚0 disk.sda1.in‚0) disk./.usagePercent‚74.36653444526075 disk.sda2.out‚0 disk.sda.out‚ 355.9609375 proc.cpu.threads‚17<†PÞ2†*- _wandbruntime‚9734  _runtime‚9734q>ömȇ* % _timestamp‚1.7884472947462523e+09 _runtime‚9734.790892919 _step‚939 & lr-AdamW/pg1‚3.308763619888644e-06 & lr-AdamW/pg2‚3.308763619888644e-06 trainer/global_step‚11449kƒ=͈*Ç _runtime‚9734.790892919 _step‚939 & lr-AdamW/pg1‚3.308763619888644e-06 & lr-AdamW/pg2‚3.308763619888644e-06 trainer/global_step‚11449 % _timestamp‚1.7884472947462523e+09«Ù*Ýe2‰*- _wandbruntime‚9734  _runtime‚9734ÿ)ñ,!"Š*œP ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1278549072205890 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1299017225107968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1277766292600832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21253559108096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1257330753926144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29092808047104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79418795 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1277766266209280 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1277766266209280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213649138 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1122135023782400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21250958898688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1299019851708928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29092757456384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1257330728498176 $ _timestamp‚1.788447294746963e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMon‰ò—itor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189212129 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169186649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79419294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1277766266209280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213649138 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1122135023782400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21250958898688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1299019851708928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29092757456384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1257330728498176 $ _timestamp‚1.788447294746963e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189212129 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169186649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79419294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1299017225107968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1277766292600832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21253559108096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1257330753926144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29092808047104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79418795 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1277766266209280 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1277874205914624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189228102 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29095315393024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1278657242723038 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1299129623756288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169200891 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169200885 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21217549337746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 _step‚941 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109802031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1277874205914624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1093134654379520 trainer/global_step‚11449 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109802028 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1278654619010194 _runtime‚9735.546896062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21252791227392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21220147622622 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21255391450112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21255391450112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1093134682513920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1277874232306176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169200885 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1122229919195136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213667372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79425596 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1122229997906944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1299129623756288 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213687432 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189227624 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1257437095100416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169200891 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1277874205914624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189228102 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29095315393024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1278657242723038 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1299129623756288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169200891 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169200885 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21217549337746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 _step‚941 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109802031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1277874205914624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1093134654379520 trainer/global_step‚11449 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109802028 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1278654619010194 _runtime‚9735.546896062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21252791227392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21220147622622 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21255391450112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21255391450112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1093134682513920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1277874232306176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169200885 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1122229919195136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213667372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79425596 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1122229997906944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1299129623756288 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213687432 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189227624 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1257437095100416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169200891 BxK—*‚(:À Ü”æÔþ†²Ë disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.56640625 disk.rbd3.in‚0% disk.nvme0n1.out‚82520.7490234375$ memory_percent‚93.65007877349854 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.in‚0* proc.memory.percent‚16.037511825561523 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.in‚ 0.56640625 disk.rbd0.in‚0 disk.rbd1.out‚4.96875 disk.sda.out‚ 494.71484375" proc.memory.rssMB‚ 2627.5859375 proc.cpu.threads‚17 network.sent‚ 2.9119307e+07 network.recv‚ 3.530064e+06' disk.nvme0c0n1.out‚82520.7490234375" disk.dm-0.out‚82520.7490234375% disk./.usageGB‚1108.3654022216797 disk.sda2.in‚0 disk.rbd2.in‚0( disk./.usagePercent‚74.3833321236124 disk.rbd3.out‚0 disk.sda3.out‚ 494.71484375# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚ 0.56640625! disk.rbd2.out‚660.87939453125) proc.memory.availableMB‚ 1040.37109375 cpu‚25.327072958603118 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 494.71484375¯2˜*- _wandbruntime‚9774  _runtime‚9774ãÐEÈ™* % _timestamp‚1.7884473343253016e+09 _runtime‚9774.369949823 _step‚943 & lr-AdamW/pg1‚2.813912481217453e-06 & lr-AdamW/pg2‚2.813912481217453e-06 trainer/global_step‚11499†£µ„Íš*Ç trainer/global_step‚11499 % _timestamp‚1.7884473343253016e+09 _runtime‚9774.369949823 _step‚943 & lr-AdamW/pg1‚2.813912481217453e-06 & lr-AdamW/pg2‚2.813912481217453e-06¯|o6N2›*- _wandbruntime‚9774  _runtime‚9774ÝJI"(œ*œP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21342589747712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169898449 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1126879654088192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1304505841224192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21342589747712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1283163251476480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21345189957120 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1304508467825152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _step‚944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1304508467825152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214564190 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1283163251476480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214564190 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1126879732813312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79757128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1304505841224192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169898455 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21309799004354 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29218035677696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29217985086976 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110252635 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1304505841224192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169898455 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21309799004354 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29218035677696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29217985086976 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110252635 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1126879654088192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1304505841224192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21342589747712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1283163251476480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21345189957120 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1304508467825152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _step‚944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1304508467825152 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214564190 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1283163251476480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214564190 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1126879732813312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79757128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668°0ýýƒ2ž*- _wandbruntime‚9775  _runtime‚9775Æy÷K£Ÿ*¦N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 384515175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110261831 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 $ _timestamp‚1.788447335075894e+09 _runtime‚9775.121082532 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1304618239872512 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1283271191181824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79764237 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1262754128386048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1097754057054720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29220487821312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚11499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190025606 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169912685 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1126974544876032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214582424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21347022299136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169912691 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21309030062354 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190026068 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1284065782161246 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21344422076416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214582424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21347022299136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169912691 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21309030062354 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190026068 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1284065782161246 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21344422076416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79764237 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1262754128386048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1097754057054720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29220487821312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚11499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190025606 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169912685 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1126974544876032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21398681444294 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1267964845925376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215479242 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215499276 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1288560263135232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21436820806144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80094579 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21434220596736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190806529 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215499276 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170610249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1288560263135232 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215479242 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21436820806144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 386089491 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1288560236743680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21434220596736 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170610249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1267964871353344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29343351747072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 _step‚948 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110712435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80094097 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1289363527369670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1309997083941376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚482 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1309994457340416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚485 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11549 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1309997083941376 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1102281071676416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29343400240640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsnÀØõëMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 190807014 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110712432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21401279728962mñ©.((¯*¢P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215499276 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170610249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1288560263135232 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215479242 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21436820806144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 386089491 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1288560236743680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21434220596736 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170610249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1267964871353344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29343351747072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 _step‚948 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110712435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80094097 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1289363527369670 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1309997083941376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚482 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1309994457340416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚485 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11549 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1309997083941376 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1102281071676416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29343400240640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 190807014 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110712432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21401279728962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21398681444294 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1267964845925376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215479242 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215499276 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1288560263135232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21436820806144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80094579 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21434220596736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190806529 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568´6ç82°*- _wandbruntime‚9814  _runtime‚98145ýδ,'±*¦N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110721628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170624485 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190822443 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21400510786962 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29345906157056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170624485 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215517536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21438653148160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215497476 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 trainer/global_step‚11549 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190822898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1102373459729920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215497476 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110721631 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1102373487864320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1131719394021376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1289471697886610 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21403109071838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1310106855988736 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 % _timestamp‚1.7884473743590326e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170624491 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1289474321599454 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29345857676800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1268071212527616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80100815 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1131719317406720 _runtime‚ 9814.40413204 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1288668176449024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21438653148160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215517536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 386121961 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170624491 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 _step‚949 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1288668202840576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1288668202840576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776ï»ÚA1'²*«N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1288668202840576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1288668202840576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110721628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170624485 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190822443 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21400510786962 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29345906157056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170624485 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215517536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21438653148160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215497476 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 trainer/global_step‚11549 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190822898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1102373459729920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215497476 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110721631 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1102373487864320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1131719394021376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1289471697886610 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21403109071838 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1310106855988736 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 % _timestamp‚1.7884473743590326e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170624491 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1289474321599454 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29345857676800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1268071212527616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80100815 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1131719317406720 _runtime‚ 9814.40413204 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1288668176449024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21438653148160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215517536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 386121961 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170624491 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 _step‚949 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0µäA~r2³*- _wandbruntime‚9814  _runtime‚9814Ï:”Œ`´*Ú 0loss/FRt2/assignment_loss‚2.1383111864328384 1loss/FBt1/assignment_loss‚0.12491336281473979 1loss/FBt2/assignment_loss‚0.22562401836815602 /loss/FBt3/detection_loss‚2.9862228706479073 trainer/global_step‚11549 /loss/FRt1/assignment_loss‚1.378958822786808 /loss/FRt4/assignment_loss‚2.527794455488523 /loss/FRt3/detection_loss‚2.8430280384953677 .loss/SRqqt3/detection_loss‚2.3155756611377 &loss/total_loss‚38.976610197706904 0loss/FRt3/assignment_loss‚3.2665515899658204 2loss/SRqqt4/assignment_loss‚1.3921251911484505 1loss/SRqqt2/detection_loss‚1.7943156020508872 /loss/FBt1/detection_loss‚0.4438755331842306 epoch‚9 2loss/SRqqt3/assignment_loss‚1.2546616186155006 /loss/FRt2/detection_loss‚2.5309923380613326 /loss/FRt4/detection_loss‚2.1898710088183484 % _timestamp‚1.7884473743609767e+09 /loss/FRt1/detection_loss‚1.8897620225790888 2loss/SRqqt1/assignment_loss‚0.8644560352812843 2loss/SRqqt2/assignment_loss‚1.0027011359524396 3loss/FBt3/assignment_loss‚0.0030858211152917647 0loss/SRqqt4/detection_loss‚1.966446358199213 /loss/FBt2/detection_loss‚2.2807718316713967 1loss/FBt4/assignment_loss‚0.10697159885603469 1loss/SRqqt1/detection_loss‚0.9284433431335186 .loss/FBt4/detection_loss‚2.521150752902031 _runtime‚9814.405551084 _step‚950°ì&„”µ*ß 0loss/FRt3/assignment_loss‚3.2665515899658204 2loss/SRqqt4/assignment_loss‚1.3921251911484505 1loss/SRqqt2/detection_loss‚1.7943156020508872 /loss/FBt1/detection_loss‚0.4438755331842306 epoch‚9 2loss/SRqqt3/assignment_loss‚1.2546616186155006 /loss/FRt2/detection_loss‚2.5309923380613326 /loss/FRt4/detection_loss‚2.1898710088183484 % _timestamp‚1.78844737436097v—#Ñ67e+09 /loss/FRt1/detection_loss‚1.8897620225790888 2loss/SRqqt1/assignment_loss‚0.8644560352812843 2loss/SRqqt2/assignment_loss‚1.0027011359524396 3loss/FBt3/assignment_loss‚0.0030858211152917647 0loss/SRqqt4/detection_loss‚1.966446358199213 /loss/FBt2/detection_loss‚2.2807718316713967 1loss/FBt4/assignment_loss‚0.10697159885603469 1loss/SRqqt1/detection_loss‚0.9284433431335186 .loss/FBt4/detection_loss‚2.521150752902031 _runtime‚9814.405551084 _step‚950 0loss/FRt2/assignment_loss‚2.1383111864328384 1loss/FBt1/assignment_loss‚0.12491336281473979 1loss/FBt2/assignment_loss‚0.22562401836815602 /loss/FBt3/detection_loss‚2.9862228706479073 trainer/global_step‚11549 /loss/FRt1/assignment_loss‚1.378958822786808 /loss/FRt4/assignment_loss‚2.527794455488523 /loss/FRt3/detection_loss‚2.8430280384953677 .loss/SRqqt3/detection_loss‚2.3155756611377 &loss/total_loss‚38.976610197706904¶ ˆ.“”¶*‚(:‰ ˜•æÔŸûŒÊ gpu.0.gpu‚92.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K îG·*‚(:¼ ˜•æÔôƒËË% disk./.usageGB‚1108.3645362854004 disk.sda.in‚ 0.57421875 disk.sda.out‚ 670.359375 disk.rbd3.out‚0$ memory_percent‚93.64888668060303 cpu‚25.32538448660075 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1778.26171875 network.sent‚ 2.9245749e+07 network.recv‚ 3.549453e+06 disk.sda3.in‚ 0.57421875 disk.dm-0.in‚ 1778.26171875 disk.rbd0.in‚0 disk.dm-1.in‚ 0.57421875 disk.dm-1.out‚ 670.359375" proc.memory.rssMB‚ 2627.5859375) disk./.usagePercent‚74.38327400990212 disk.rbd1.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚82717.16455078125* proc.memory.percent‚16.037511825561523) proc.memory.availableMB‚ 1040.56640625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚82717.16455078125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚662.6337890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚4.96875 disk.sda1.in‚0! disk.nvme0n1.in‚ 1778.26171875& disk.nvme0n1.out‚82717.16455078125 disk.sda3.out‚ 670.359375ƒE“¸*‚(:ˆ §•æÔ¯†ðÈ gpu.0.gpu‚93.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚95.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ ¢ßJ¹*‚(:¿ §•æÔ¥ÞƒÌ* proc.memory.percent‚16.037511825561523) disk./.usagePercent‚74.38325864945007 disk.dm-1.in‚ 0.57421875 disk.sda1.out‚0# disk.dm-0.out‚82717.45361328125! disk.nvme0n1.in‚ 1778.26171875& disk.nvme0n1.out‚82717.45361328125 disk.rbd2.in‚0 disk.rbd3.in‚0 cpu‚25.32493819023902 disk.rbd0.out‚0 disk.sda.in‚ 0.57421875 disk.sda1.in‚0$ memory_percent‚93.64817142486572) proc.memory.availableMB‚ 1040.68359375 disk.sda3.in‚ 0.57421875 disk.sda3.out‚ 670.8984375 disk.drbd1777.in‚0 disk.rbd1.out‚4.96875 disk.sda2.in‚0( disk.nvme0c0n1.out‚82717.45361328125 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.555436e+06 disk.dm-1.out‚ 670.8984375 disk.sda.out‚ 670.8984375 disk.rbd3.out‚0" proc.memory.rssMB‚ 2627.5859375 network.sent‚ 2.9355036e+07 disk.dm-0.in‚ 1778.26171875 disk.sda2.out‚0% disk./.usageGB‚1108.3643074035645 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.rbd2.out‚662.7119140625fñtU2º*- _wandbruntime‚9853  _runtime‚9853.W¦È»* % _timestamp‚1.7884474131857872e+09 _runtime‚9853.230493004 _step‚951 & lr-AdamW/pg1‚1.943681826603238e-06 & lr-AdamW/pg2‚1.943681826603238e-06 trainer/global_step‚115995äÇͼ*Ç trainer/global_step‚11599 % _timestamp‚1.7884474131857872e+09 _runtime‚9853.230493004 _step‚951 & lr-AdamW/pg1‚1.943681826603238e-06 & lr-AdamW/pg2‚1.943681826603238e-06·0O2½*- _wandbruntime‚9853  _runtime‚9853`ÿk›#(¾*P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191601230 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315483073456640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1294772066807878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 % _timestamp‚1.7884474131868923e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21525851445760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171322055 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1136369213954560 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293957248402432 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171322055 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216394294 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21528451655168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171322049 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216414328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216414328 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1106900502486016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 _step‚952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚9853.231998011 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293957222010880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315485700057600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21525851445760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111172235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191601695 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80428998 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1273281930066944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315483073456640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216394294 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21528451655168 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1136369288485376 trainer/global_step‚11599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293957222010880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1273281904638976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736383 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376þ-/((¿*¢P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111172232 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚74530816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29468739602944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1273281904638976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387736383 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191601230 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315483073456640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1294772066807878 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 % _timestamp‚1.7884474131868923e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21525851445760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171322055 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1136369213954560 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293957248402432 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171322055 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216394294 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21528451655168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171322049 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216414328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216414328 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1106900502486016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 _step‚952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚9853.231998011 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293957222010880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚46396416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315485700057600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21525851445760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111172235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191601695 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80428998 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1273281930066944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315483073456640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216394294 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21528451655168 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1136369288485376 trainer/global_step‚11599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293957222010880 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚953 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1136464202136576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1315595472104960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1315595472104960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748813 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 9853.98386439 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216412528 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1294065161716224 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111181431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚11599 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21530283997184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216432588 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1294882861037662 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1294065161716224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171336285 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191616911 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111181428 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.o«…†‘… n_train_batch_end/allocated_bytes.small_pool.freed‚21527683774464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21494589796446 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21530283997184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 $ _timestamp‚1.788447413938587e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29471263116800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21527683774464 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216412528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1273388271241216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21491991511570 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80435483 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111181431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚11599 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21530283997184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216432588 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1294882861037662 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1294065161716224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171336285 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191616911 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111181428 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21527683774464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21494589796446 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21530283997184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 $ _timestamp‚1.788447413938587e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29471263116800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21527683774464 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216412528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1273388271241216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21491991511570 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80435483 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚953 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1136464202136576 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1315595472104960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1315595472104960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748813 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 9853.98386439¹šª2Ã*- _wandbruntime‚9853  _runtime‚9853–AT\Ä*Ö /loss/FBt4/detection_loss‚2.3345864878760443 $ _timestamp‚1.788447413940795e+09 _runtime‚9853.985402369 2loss/SRqqt2/assignment_loss‚0.8890780325786307 2loss/SRqqt3/assignment_loss‚1.4197246211317733 0loss/FBt4/assignment_loss‚0.1612209165468812 /loss/FBt3/detection_loss‚2.4943542913957075 2loss/SRqqt4/assignment_loss‚0.9758677783122819 0loss/SRqqt1/detection_loss‚1.021398769307231 1loss/SRqqt4/detection_loss‚1.8651933996006846 /loss/FBt1/detection_loss‚0.5463595384568353 0loss/FBt1/assignment_loss‚0.1477439774186197 1loss/SRqqt3/detection_loss‚1.9452939130689786 /loss/FBt2/detection_loss‚2.1322753700343045 trainer/global_step‚11599 0loss/FRt4/assignment_loss‚2.0756631032987074 .loss/FRt3/detection_loss‚2.424958477768679 epoch‚9 _step‚954 /loss/FRt2/assignment_loss‚2.091178127025303 1loss/FBt2/assignment_loss‚0.00538360839180479 /loss/FRt1/detection_loss‚2.0319713996723294 /loss/FRt4/detection_loss‚2.3252064592458983 0loss/FRt1/assignment_loss‚1.2694755761246932 %loss/total_loss‚36.15261794130529 2loss/SRqqt1/assignment_loss‚0.7970669953566458 1loss/FBt3/assignment_loss‚0.17340461862940257 /loss/FRt3/assignment_loss‚2.955751129361086 /loss/FRt2/detection_loss‚2.3592910256824995 0loss/SRqqt2/detection_loss‚1.710170325020264/œåaÅ*Û /loss/FBt3/detection_loss‚2.4943542913957075 2loss/SRqqt4/assignment_loss‚0.9758677783122819 0loss/SRqqt1/detection_loss‚1.021398769307231 1loss/SRqqt4/detection_loss‚1.8651933996006846 /loss/FBt1/detection_loss‚0.5463595384568353 0loss/FBt1/assignment_loss‚0.1477439774186197 1loss/SRqqt3/detection_loss‚1.9452939130689786 /loss/FBt2/detection_loss‚2.1322753700343045 trainer/global_step‚11599 0loss/FRt4/assignment_loss‚2.0756631032987074 .loss/FRt3/detection_loss‚2.424958477768679 epoch‚9 _step‚954 /loss/FRt2/assignment_loss‚2.091178127025303 1loss/FBt2/assignment_loss‚0.00538360839180479 /loss/FRt1/detection_loss‚2.0319713996723294 /loss/FRt4/detection_loss‚2.3252064592458983 0loss/FRt1/assignment_loss‚1.2694755761246932 %loss/total_loss‚36.15261794130529 2loss/SRqqt1/assignment_loss‚0.7970669953566458 1loss/FBt3/assignment_loss‚0.17340461862940257 /loss/FRt3/assignment_loss‚2.955751129361086 /loss/FRt2/detection_loss‚2.3592910256824995 0loss/SRqqt2/detection_loss‚1.710170325020264 /loss/FBt4/detection_loss‚2.3345864878760443 $ _timestamp‚1.788447413940795e+09 _runtime‚9853.985402369 2loss/SRqqt2/assignment_loss‚0.8890780325786307 2loss/SRqqt3/assignment_loss‚1.4197246211317733 0loss/FBt4/assignment_loss‚0.1612209165468812º^wqPŸÆ*‚(:” ¶•æÔø¢•É gpu.0.gpu‚87.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<r!FÇ*‚(:» ¶•æÔËãÂË disk.sda.in‚ 0.57421875% disk./.usageGB‚1108.3643074035645 disk.sda3.in‚ 0.57421875# disk.nvme0c0n1.in‚ 1778.26171875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 671.6328125 disk.drbd1777.out‚0 disk.rbd2.out‚662.7119140625 disk.sda2.out‚0( proc.memory.availableMB‚ 1040.6953125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚82815.1533203125 disk.dm-0.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.drbd1777.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚82815.1533203125 disk.rbd0.in‚0 disk.sda.out‚ 671.6328125 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1778.26171875 network.sent‚ 2.9359403e+07 disk.rbd1.out‚4.96875 disk.sda1.in‚0" proc.memory.rssMB‚ 2627.5859375 cpu‚25.32452349551618 proc.cpu.threads‚17 network.recv‚ 3.559041e+06) disk./.usagePercent‚74.38325864945007" disk.dm-0.out‚82815.1533203125 disk.dm-1.out‚ 671.6328125 disk.rbd3.in‚0$ memory_percent‚93.64809989929199* proc.memory.percent‚16.037511825561523 disk.rbd1.in‚0ÏÏp“È*‚(:ˆ Å•æÔ¸ž‘É gpu.0.gpu‚65.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚261.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.307 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª= 4DÉ*‚(:¹ Å•æÔÖ‘ÛÊ% disk.nvme0n1.out‚82815.6318359375* proc.memory.percent‚16.037511825561523 network.sent‚ 2.9468688e+07 disk.drbd1777.in‚0" disk.dm-0.out‚82815.6318359375 disk.dm-1.in‚ 0.57421875 disk.dm-1.out‚ 672.078125 disk.rbd1.in‚0 disk.sda.out‚ 672.078125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚82815.6318359375% disk./.usageGB‚1108.3643074035645 disk.rbd2.in‚0 disk.sda2.in‚0# memory_percent‚93.6485767364502( proc.memory.availableMB‚ 1040.6171875 disk.rbd1.out‚4.96875! disk.rbd2.out‚662.87646484375! disk.nvme0n1.in‚ 1778.26171875 proc.cpu.threads‚17 network.recv‚ 3.565692e+06 disk.rbd3.out‚0) disk./.usagePercent‚74.38325864945007 disk.sda1.out‚0 cpu‚25.324086478677156# disk.nvme0c0n1.in‚ 1778.26171875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 672.078125 disk.dm-0.in‚ 1778.26171875" proc.memory.rssMB‚ 2627.5859375 disk.sda.in‚ 0.57421875d[¾”Ê*‚(:‰ Ô•æÔ× ÞÈ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚228.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.6796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨kÆ'@Ë*‚(:µ Ô•æÔ·ˆ÷Ë) disk./.usagePercent‚74.38325864945007 disk.rbd3.in‚0 disk.sda3.out‚ 672.40625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.out‚ 672.40625# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.rbd1.out‚4.96875! disk.rbd2.out‚662.87646484375 disk.sda2.out‚0 disk.rbd3.out‚0! disk.dm-0.out‚82913.357421875 disk.dm-1.out‚ 672.40625 disk.rbd1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.570088e+06" proc.memory.rssMB‚ 2627.5859375! disk.nvme0n1.in‚ 1778.26171875$ disk.nvme0n1.out‚82913.357421875 disk.rbd0.in‚0 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.in‚0 disk.sda.in‚ 0.57421875) proc.memory.availableMB‚ 1040.51953125 network.sent‚ 2.9474565e+07% disk./.usageGB‚1108.3643074035645 disk.sda3.in‚ 0.57421875 disk.drbd1777.out‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚82913.357421875 disk.sda2.in‚0$ memory_percent‚93.64917278289795* proc.memory.percent‚16.037511825561523 cpu‚25.323690487161915nOE42Ì*- _wandbruntime‚9892  _runtime‚9892>¨íÊÍ*Ä ' lr-AdamW/pg1‚1.5684836654242928e-06 ' lr-AdamW/pg2‚1.5684836654242928e-06 trainer/global_step‚11649 % _timestamp‚1.7884474521644802e+09 _runtime‚9892.209126795 _step‚955ÁïÎxÏÎ*É ' lr-AdamW/pg1‚1.5684836654242928e-06 ' lr-AdamW/pg2‚1.5684836654242928e-06 trainer/global_step‚11649 % _timestamp‚1.7884474521644802e+09 _runtime‚9892.209126795 _step‚955»8Ü~2Ï*- _wandbruntime‚9892  _runtime‚9892¦?3‘#(Ð*P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172033849 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192396625 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1299354233669632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21620082504192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192397101 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111632032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1320974316173824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278598988780544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21581642893510 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _step‚956 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343195 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1320971689572864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111632035 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1141114129664000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80764593 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1300183229958722 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 % _timestamp‚1.7884474521651776e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1141114206291968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21620082504192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1320971689572864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217309346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21617482294784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217329380 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1111519905161216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1300180606246086 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172033849 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1299354207278080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1299354207278080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [HDeviceSta`vzÐótsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21581642893510 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 _step‚956 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343195 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1320971689572864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111632035 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1141114129664000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚80764593 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1300183229958722 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 % _timestamp‚1.7884474521651776e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1141114206291968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21620082504192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1320971689572864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217309346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21617482294784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217329380 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1111519905161216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48493568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1300180606246086 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172033849 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1299354207278080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1299354207278080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172033849 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192396625 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¼å/˜2Ò*- _wandbruntime‚9892  _runtime‚9892ÉSœù,'Ó*¦N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884474528989122e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80771254 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1111612265080320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1141209107555840 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1278705329954816 _step‚957 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172048091 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21621914846208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 9892.94414086 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217327580 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21619314623488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1141209030941184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1300288776763026 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1299462173374976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217327580 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192412482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21583472236178 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217347640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29596814341120 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21619314623488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29596765860864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172048091 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1278705304526848 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1299462146983424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21621914846208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80771707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395731 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1299462146983424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172048085 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192412938 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21583472236178 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217347640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29596814341120 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21619314623488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29596765860864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172048091 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1278705304526848 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1299462146983424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21621914846208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80771707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395731 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1299462146983424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172048085 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 192412938 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884474528989122e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80771254 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1111612265080320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1141209107555840 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1278705329954816 _step‚957 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172048091 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21621914846208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 9892.94414086 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217327580 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21619314623488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1141209030941184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1300288776763026 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1299462173374976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217327580 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192412482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400½ÑE‚2Õ*- _wandbruntime‚9892  _runtime‚9892¼ÖŸ\Ö*Ö 1loss/SRqqt2/detection_loss‚1.5964271236135035 0loss/SRqqt4/detection_loss‚1.660123963330714 .loss/FBt3/detection_loss‚2.684971880912781 1loss/SRqqt1/detection_loss‚0.9983279471527096 /loss/FBt4/detection_loss‚2.0812883214517073 1loss/FBt2/assignment_loss‚0.08962661861005472 1loss/FBt4/assignment_loss‚0.11930283216034612 /loss/FBt1/detection_loss‚0.3894020915807535 _runtime‚9892.945794105 _step‚958 2loss/SRqqt2/assignment_loss‚1.1287761752930832 1loss/FBt1/assignment_loss‚0.19479757763429587 trainer/global_step‚11649 epoch‚9 0loss/FRt2/assignment_loss‚2.1290298511111545 .loss/FRt4/detection_loss‚2.100522171796822 /loss/FRt3/assignment_loss‚3.328801029437297 /loss/FRt4/assignment_loss‚2.366336309328312 2loss/SRqqt3/assignment_loss‚1.2435164949191468 .loss/FBt2/detection_loss‚2.877449711163839 &loss/total_loss‚37.037564757731914 $ _timestamp‚1.788447452901122e+09 0loss/FRt1/assignment_loss‚1.2803382600347202 2loss/SRqqt4/assignment_loss‚1.3952160505226479 0loss/FBt3/assignment_loss‚0.3538630512543023 /loss/FRt3/detection_loss‚2.5152024791962972 1loss/SRqqt3/detection_loss‚2.4080350271293094 2loss/SRqqt1/assignment_loss‚0.6950484772592231 /loss/FRt1/detection_loss‚1.4075220463952671 .loss/FRt2/detection_loss‚1.9936392664436319ÿr®a×*Û /loss/FRt1/detection_loss‚1.4075220463952671 .loss/FRt2/detection_loss‚1.993639266443631 1loss/SRqqt2/detection_loss‚1.5964271236135035 0loss/SRqqt4/detection_loss‚1.660123963330714 .loss/FBt3/detection_loss‚2.684971880912781 1loss/SRqqt1/detection_loss‚0.9983279471527096 /loss/FBt4/detection_loss‚2.0812883214517073 1loss/FBt2/assignment_loss‚0.08962661861005472 1loss/FBt4/assignment_loss‚0.11930283216034612 /loss/FBt1/detection_loss‚0.3894020915807535 _runtime‚9892.945794105 _step‚958 2loss/SRqqt2/assignment_loss‚1.1287761752930832 1loss/FBt1/assignment_loss‚0.19479757763429587 trainer/global_step‚11649 epoch‚9 0loss/FRt2/assignment_loss‚2.1290298511111545 .loss/FRt4/detection_loss‚2.100522171796822 /loss/FRt3/assignment_loss‚3.328801029437297 /loss/FRt4/assignment_loss‚2.366336309328312 2loss/SRqqt3/assignment_loss‚1.2435164949191468 .loss/FBt2/detection_loss‚2.877449711163839 &loss/total_loss‚37.037564757731914 $ _timestamp‚1.788447452901122e+09 0loss/FRt1/assignment_loss‚1.2803382600347202 2loss/SRqqt4/assignment_loss‚1.3952160505226479 0loss/FBt3/assignment_loss‚0.3538630512543023 /loss/FRt3/detection_loss‚2.5152024791962972 1loss/SRqqt3/detection_loss‚2.4080350271293094 2loss/SRqqt1/assignment_loss‚0.6950484772592231¾Ž¦‘òŸØ*‚(:” ã•æÔžô„É gpu.0.gpu‚93.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚256.466' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.293200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÃIÙ*‚(:¾ ã•æÔѤšË( disk.nvme0c0n1.out‚82917.51025390625 disk.dm-1.in‚ 0.57421875 cpu‚25.323255552048067 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚82917.51025390625 disk.sda.out‚ 673.296875# proc.memory.rssMB‚ 2629.82421875 network.sent‚ 2.9480847e+07) disk./.usagePercent‚74.38349776048705 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.sda.in‚ 0.57421875( proc.memory.availableMB‚ 1039.9296875* proc.memory.percent‚16.051173210144043! disk.nvme0n1.in‚ 1778.26171875& disk.nvme0n1.out‚82917.51025390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚4.96875! disk.rbd2.out‚664.59130859375 proc.cpu.threads‚17 disk.rbd3.in‚0$ memory_percent‚93.65277290344238% disk./.usageGB‚1108.3678703308105 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.57421875# disk.nvme0c0n1.in‚ 1778.26171875 network.recv‚ 3.574679e+06 disk.dm-1.out‚ 673.296875 disk.rbd2.in‚0 disk.sda3.out‚ 673.296875 disk.sda2.in‚0ðð6aÚ*‚(:“ ò•æÔÿÌÞÈ gpu.0.gpu‚88.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.N<ß«=198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚94.462' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.892400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0."DsMÛ*‚(: ò•æÔ榃Ì disk.sda2.out‚0 disk.rbd3.in‚0) disk./.usagePercent‚74.38349801649458 disk.rbd0.in‚0 disk.dm-1.in‚ 0.57421875 disk.drbd1777.out‚0 disk.rbd1.out‚4.96875 disk.sda.out‚ 673.7578125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚93.64907741546631 proc.cpu.threads‚17 network.recv‚ 3.580128e+06 disk.sda3.out‚ 673.7578125# disk.nvme0c0n1.in‚ 1778.26171875! disk.rbd2.out‚664.59130859375 disk.dm-1.out‚ 673.7578125 disk.rbd1.in‚0 cpu‚25.322813819712238 disk.sda3.in‚ 0.57421875 disk.sda1.in‚0& disk.nvme0n1.out‚83015.30712890625* proc.memory.percent‚16.051173210144043 network.sent‚ 2.9589244e+07% disk./.usageGB‚1108.3678741455078( disk.nvme0c0n1.out‚83015.30712890625 disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚83015.30712890625 disk.sda.in‚ 0.57421875 disk.rbd3.out‚0) proc.memory.availableMB‚ 1040.53515625 disk.drbd1777.in‚0 disk.rbd2.in‚0! disk.nvme0n1.in‚ 1778.26171875# proc.memory.rssMB‚ 2629.82421875ÎR8“Ü*‚(:ˆ –æÔ­˜“É gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚94.572' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sc:'LÝ*‚(:Á –æÔžø‹Ë disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 664.693359375 disk.sda2.out‚0 disk.dm-1.in‚ 0.57421875 network.recv‚ 3.58424e+06 disk.sda.in‚ 0.57421875 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚83015.71630859375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.rbd1.out‚4.96875 disk.dm-0.in‚ 1778.26171875 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1778.26171875$ memory_percent‚93.64936351776123# proc.memory.rssMB‚ 2629.82421875 cpu‚25.32240770520939) disk./.usagePercent‚74.38350032056239 disk.rbd2.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1040.48828125 disk.drbd1777.out‚0 disk.sda.out‚ 674.41015625 disk.rbd3.in‚0 disk.sda3.in‚ 0.57421875 disk.dm-1.out‚ 674.41015625* proc.memory.percent‚16.051173210144043 network.sent‚ 2.9595159e+07 disk.sda1.in‚0 disk.sda3.out‚ 674.41015625( disk.nvme0c0n1.out‚83015.71630859375# disk.dm-0.out‚83015.71630859375 proc.cpu.threads‚17% disk./.usageGB‚1108.3679084777832r9¨r2Þ*- _wandbruntime‚9931  _runtime‚9931Î3„Êß*Ä % _timestamp‚1.7884474914572341e+09 _runtime‚9931.501956145 _step‚959 ' lr-AdamW/pg1‚1.2333373127906904e-06 ' lr-AdamW/pg2‚1.2333373127906904e-06 trainer/global_step‚11699rfÏà*É trainer/global_step‚11699 % _timestamp‚1.7884474914572341e+09 _runtime‚9931.501956145 _step‚959 ' lr-AdamW/pg1‚1.2333373127906904e-06 ' lr-AdamW/pg2‚1.2333373127906904e-06¿Q÷”»2á*- _wandbruntime‚9931  _runtime‚9931Î 1é#(â*P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1304751218936832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81100421 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218244432 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1326462932290048 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1116139307836416 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29719457651200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1304751192545280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1304751218936832 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1326462932290048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172745649 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21709113143808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112091835 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _step‚960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1326460305689088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚493 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1326460305689088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29719407060480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884474914580455e+09 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81099928 _runtime‚9931.502887218 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1283916022066176 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193191760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1304751192545280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172745655 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21711713353216¿åøW((ã*¢P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21709113143808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172745649 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218224398 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21711713353216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21675721902786 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172745655 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1283916047494144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390990087 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1305589145684294 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112091835 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 _step‚960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1326460305689088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚493 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1326460305689088 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29719407060480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884474914580455e+09 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81099928 _runtime‚9931.502887218 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1283916022066176 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193191760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1304751192545280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172745655 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21711713353216 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1304751218936832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81100421 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218244432 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1326462932290048 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1116139307836416 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29719457651200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1304751192545280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1304751218936832 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1326462932290048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172745649 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21709113143808 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193207664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1305697316201234 % _timestamp‚1.7884474922000782e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391002517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1305699939914078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11699 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112101031 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1116231667755520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21677551245662 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21713545695232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1326572704337408 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1304859158642176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218242632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172759885 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81107093 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1304859158642176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21710945472512 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391022583 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21674952960786 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172759891 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218262692 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1116231695889920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29721908492288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1145953576247808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172759891 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218262692 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1116231695889920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚29721908492288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1145953576247808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193207664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1305697316201234 % _timestamp‚1.7884474922000782e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391002517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1305699939914078 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11699 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112101031 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1116231667755520 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21677551245662 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21713545695232ÁÉ2ç*- _wandbruntime‚9932  _runtime‚99324”šý`è*Ú /loss/FRt1/detection_loss‚1.6254553927362643 /loss/FRt2/detection_loss‚2.4168530073430805 /loss/FRt4/detection_loss‚2.0795813745006604 2loss/SRqqt1/assignment_loss‚0.8831129375099347 1loss/SRqqt2/detection_loss‚1.7761826662624136 2loss/SRqqt4/assignment_loss‚1.2944412511812213 .loss/FBt4/detection_loss‚2.041120520896382 trainer/global_step‚11699 % _timestamp‚1.7884474922022238e+09 0loss/FBt1/detection_loss‚0.44572968553849174 %loss/total_loss‚37.09964578405692 _runtime‚9932.246818753 2loss/SRqqt3/assignment_loss‚1.6397010218740131 1loss/FBt2/assignment_loss‚0.10265547975521055 0loss/FBt4/assignment_loss‚0.2678206810055094 /loss/FRt3/detection_loss‚2.4492516159870936 0loss/FRt1/assignment_loss‚1.1963000324639408 0loss/FRt4/assignment_loss‚2.0072324314015977 /loss/SRqqt3/detection_loss‚2.37952678501606 /loss/FBt3/detection_loss‚2.9086363116900125 _step‚962 1loss/FBt1/assignment_loss‚0.07962191592197491 1loss/SRqqt1/detection_loss‚1.0861990995452566 1loss/SRqqt4/detection_loss‚1.7637033043429255 .loss/FBt2/detection_loss‚2.102519883049859 epoch‚9 0loss/FRt2/assignment_loss‚2.5828919053077697 0loss/FRt3/assignment_loss‚2.8913610135355303 2loss/SRqqt2/assignment_loss‚0.9947896634248343 1loss/FBt3/assignment_loss‚0.08495780376688344ÞšÐeé*ß %loss/total_loss‚37.09964578405692 _runtime‚9932.246818753 2loss/SRqqt3/assignment_loss‚1.6397010218740131 1loss/FBt2/assignment_loss‚0.10265547975521055 0loss/FBt4/assignment_loss‚0.2678206810055094 /loss/FRt3/detection_loss‚2.4492516159870936 0loss/FRt1/assignment_loss‚1.1963000324639408 0loss/FRt4/assignment_loss‚2.0072324314015977 /loss/SRqqt3/detection_loss‚2.37952678501606 /loss/FBt3/detection_loss‚2.9086363116900125 _step‚962 1loss/FBt1/assignment_loss‚0.07962191592197491 1loss/SRqqt1/detection_loss‚1.0861990995452566 1loss/SRqqt4/detection_loss‚1.7637033043429255 .loss/FBt2/detection_loss‚2.102519883049859 epoch‚9 0loss/FRt2/assignment_loss‚2.5828919053077697 0loss/FRt3/assignment_loss‚2.8913610135355303 2loss/SRqqt2/assignment_loss‚0.9947896634248343 1loss/FBt3/assignment_loss‚0.08495780376688344 /loss/FRt1/detection_loss‚1.6254553927362643 /loss/FRt2/detection_loss‚2.4168530073430805 /loss/FRt4/detection_loss‚2.0795813745006604 2loss/SRqqt1/assignment_loss‚0.8831129375099347 1loss/SRqqt2/detection_loss‚1.7761826662624136 2loss/SRqqt4/assignment_loss‚1.2944412511812213 .loss/FBt4/detection_loss‚2.041120520896382 trainer/global_step‚11699 % _timestamp‚1.7884474922022238e+09 0loss/FBt1/detection_loss‚0.44572968553849174Âå̓؞ê*‚(:“ –æÔ¾ÂýÈ gpu.0.gpu‚68.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚41.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ýF Më*‚(: –æÔöáË disk.sda1.out‚0 disk.sda3.in‚ 0.57421875 disk.sda3.out‚ 674.76171875 disk.dm-1.in‚ 0.57421875 network.sent‚ 2.9601053e+07 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚83016.47216796875 disk.rbd0.out‚0 disk.dm-0.in‚ 1778.26171875 network.recv‚ 3.588696e+06 disk.sda2.out‚0( disk.nvme0c0n1.out‚83016.47216796875) proc.memory.availableMB‚ 1040.48046875) disk./.usagePercent‚74.38348828820828 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 674.76171875$ memory_percent‚93.64941120147705# proc.memory.rssMB‚ 2629.82421875* proc.memory.percent‚16.051173210144043 cpu‚25.32196932260553 disk.sda.in‚ 0.57421875! disk.nvme0n1.in‚ 1778.26171875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875# disk.dm-0.out‚83016.47216796875 proc.cpu.threads‚17% disk./.usageGB‚1108.3677291870117 disk.rbd1.in‚0 disk.rbd1.out‚4.96875 disk.sda.out‚ 674.76171875 disk.rbd2.out‚ 664.693359375 disk.sda1.in‚0Ë6ìÆ”ì*‚(:‰ Ÿ–æÔŹêÊ gpu.0.gpu‚71.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚203.631' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.7262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íS­ãDí*‚(:¹ Ÿ–æÔœœŠÍ network.sent‚ 2.9710396e+07% disk./.usageGB‚1108.3678016662598& disk.nvme0n1.out‚83113.93310546875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.out‚ 675.34375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1778.26171875 disk.dm-1.out‚ 675.34375 disk.drbd1777.in‚0 disk.rbd1.out‚4.96875 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚83113.93310546875 disk.dm-1.in‚ 0.57421875) disk./.usagePercent‚74.38349315235143 disk.rbd1.in‚0 disk.rbd2.out‚ 664.771484375 disk.sda.in‚ 0.57421875# disk.nvme0c0n1.in‚ 1778.26171875) proc.memory.availableMB‚ 1040.11328125# proc.memory.rssMB‚ 2629.82421875* proc.memory.percent‚16.051173210144043! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.out‚ 675.34375# memory_percent‚93.6516523361206 cpu‚25.321578944273874 network.recv‚ 3.595356e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚83113.93310546875 disk.drbd1777.out‚0Û‘ìd2î*- _wandbruntime‚9970  _runtime‚9970×i…3Çï*Á $ _timestamp‚1.788447530521355e+09 _runtime‚9970.566008056 _step‚963 & lr-AdamW/pg1‚9.383126127000052e-07 & lr-AdamW/pg2‚9.383126127000052e-07 trainer/global_step‚11749wï»Ìð*Æ & lr-AdamW/pg1‚9.383126127000052e-07 & lr-AdamW/pg2‚9.383126127000052e-07 trainer/global_step‚11749 $ _timestamp‚1.788447530521355e+09 _runtime‚9970.566008056 _step‚963ÃCáÞG2ñ*- _wandbruntime‚9970  _runtime‚9970²f®Ð#(ò*P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81436742 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193988377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21764604342726 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219159484 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 _step‚964 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21803344202240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1289233080779776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219139450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219139450 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112551632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173457455 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1120758710511616 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29844708722688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1120758682377216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112551635 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29844658131968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173457449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21800743992832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1310148204204032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1331951548406272 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81436262 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173457455 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1331948921805312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚483 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193987894 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 trainer/global_step‚11749 _runtime‚9970.567159414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1150603340509184 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884475305221868e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392596899 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219159484 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21803344202240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392616939 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1331948921805312 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1310148177812480 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1150603419234304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173457449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1311000308835138 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1310148177812480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310997685122502 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21767202627394 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1310148204204032 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1331951548406272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21800743992832 Žh•-ó*¢P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173457455 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1331948921805312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚483 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193987894 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 trainer/global_step‚11749 _runtime‚9970.567159414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1150603340509184 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 % _timestamp‚1.7884475305221868e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392596899 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219159484 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21803344202240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392616939 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1331948921805312 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1310148177812480 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1150603419234304 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173457449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 394690560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1311000308835138 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1310148177812480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310997685122502 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21767202627394 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1310148204204032 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1331951548406272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.ä;5ûsmall_pool.freed‚21800743992832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81436742 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193988377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21764604342726 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219159484 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 _step‚964 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21803344202240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1289233080779776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219139450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219139450 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112551632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173457455 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1120758710511616 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚29844708722688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1120758682377216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112551635 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29844658131968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173457449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21800743992832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1310148204204032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2711634944 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1331951548406272 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚81436262ÄsŠ2ô*- _wandbruntime‚9971  _runtime‚9971RÜi_,'õ*¦N QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219177744 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219157684 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81442932 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1120851098565120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1310256117517824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1311105855639442 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173471685 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1332061320453632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21766433685394 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21769031970270 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1150698319815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21805176544256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29847221250560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112560831 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1289339421954048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚9971.308901388 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81443392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219157684 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173471685 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194004223 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21802576321536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1150698241103872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194003760 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1120851070430720 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311108479352286 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575û"¹1'ö*«N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81442932 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1120851098565120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1310256117517824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1311105855639442 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173471685 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1332061320453632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21766433685394 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21769031970270 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1150698319815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21805176544256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29847221250560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112560831 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1289339421954048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 _runtime‚9971.308901388 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81443392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219157684 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173471685 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194004223 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21802576321536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1150698241103872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194003760 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1120851070430720 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311108479352286 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392649435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219177744 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219157684 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776Å{ŒÀ2÷*- _wandbruntime‚9971  _runtime‚9971ÂùÀ[ø*Õ 2loss/SRqqt1/assignment_loss‚0.7413418501305084 2loss/SRqqt4/assignment_loss‚1.1046075254286591 /loss/FBt4/detection_loss‚2.6019698002121667 epoch‚9 _step‚966 .loss/FRt1/detection_loss‚1.545318909265377 .loss/FRt3/detection_loss‚1.953749871209126 .loss/FBt2/detection_loss‚2.857014834880829 /loss/FBt3/detection_loss‚3.4926814834276834 trainer/global_step‚11749 $ _timestamp‚1.788447531265351e+09 1loss/FBt3/assignment_loss‚0.30675269005587325 0loss/FRt1/assignment_loss‚1.2167846361796062 .loss/FRt2/assignment_loss‚2.37322836413103 2loss/SRqqt3/assignment_loss‚1.2947073632173918 .loss/FRt4/detection_loss‚2.470595558715421 1loss/SRqqt3/detection_loss‚1.9362504570321604 0loss/FBt2/assignment_loss‚0.3211202358826995 /loss/FRt4/assignment_loss‚2.162646231263183 1loss/FBt1/assignment_loss‚0.12637690708786112 1loss/SRqqt2/detection_loss‚1.7058072943164377 %loss/total_loss‚38.25810725075685 1loss/FBt4/assignment_loss‚0.11887466338380777 0loss/SRqqt4/detection_loss‚2.109221436279385 2loss/SRqqt2/assignment_loss‚1.0735654320035661 1loss/SRqqt1/detection_loss‚0.9197510915497938 0loss/FBt1/detection_loss‚0.42681740237296456 0loss/FRt3/assignment_loss‚3.1146007224365517 _runtime‚9971.309923503 /loss/FRt2/detection_loss‚2.2843224902947745?ô`ù*Ú /loss/FBt3/detection_loss‚3.4926814834276834 trainer/global_step‚11749 $ _timestamp‚1.788447531265351e+09 1loss/FBt3/assignment_loss‚0.30675269005587325 0loss/FRt1/assignment_loss‚1.2167846361796062 .loss/FRt2/assignment_loss‚2.37322836413103 2loss/SRqqt3/assignment_loss‚1.2947073632173918 .loss/FRt4/detection_loss‚2.470595558715421 1loss/SRqqt3/detection_loss‚1.9362504570321604 0loss/FBt2/assignment_loss‚0.3211202358826995 /loss/FRt4/assignment_loss‚2.162646231263183 1loss/FBt1/assignment_loss‚0.12637690708786112 1loss/SRqqt2/detection_loss‚1.7058072943164377 %loss/total_loss‚38.25810725075685 1loss/FBt4/assignment_loss‚0.11887466338380777 0loss/SRqqt4/detection_loss‚2.109221436279385 2loss/SRqqt2/assignment_loss‚1.0735654320035661 1loss/SRqqt1/detection_loss‚0.9197510915497938 0loss/FBt1/detection_loss‚0.42681740237296456 0loss/FRt3/assignment_loss‚3.1146007224365517 _runtime‚9971.309923503 /loss/FRt2/detection_loss‚2.2843224902947745 2loss/SRqqt1/assignment_loss‚0.7413418501305084 2loss/SRqqt4/assignment_loss‚1.1046075254286591 /loss/FBt4/detection_loss‚2.6019698002121667 epoch‚9 _step‚966 .loss/FRt1/detection_loss‚1.545318909265377 .loss/FRt3/detection_loss‚1.953749871209126 .loss/FBt2/detection_loss‚2.857014834880829ÆŠ:µ­žú*‚(:“ ®–æÔ›—íÊ gpu.0.gpu‚88.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚234.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚46.97559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ꡉôHû*‚(:½ ®–æÔ»»™Ë disk.rbd3.out‚0* proc.memory.percent‚16.051173210144043 proc.cpu.threads‚17) disk./.usagePercent‚74.38352438527062 disk.rbd2.out‚ 664.771484375 disk.sda2.out‚0 disk.sda3.out‚ 675.6640625# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2629.82421875 network.sent‚ 2.9715678e+07 network.recv‚ 3.599258e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚83114.59521484375 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1778.26171875 disk.rbd1.in‚0$ memory_percent‚93.64862442016602' proc.memory.availableMB‚ 1040.609375 cpu‚25.32115658256707 disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚83114.59521484375 disk.drbd1777.out‚0 disk.sda.in‚ 0.57421875 disk.sda1.out‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.out‚0 disk.dm-1.out‚ 675.6640625 disk.rbd3.in‚0% disk./.usageGB‚1108.3682670593262& disk.nvme0n1.out‚83114.59521484375 disk.rbd1.out‚4.96875 disk.sda.out‚ 675.6640625Ip €”ü*‚(:‰ ½–æÔñ·–É gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚42.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ja(Ký*‚(:À ½–æÔ´ËÅË disk.dm-0.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.drbd1777.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.57421875 disk.sda2.in‚0 disk.sda3.out‚ 676.2421875# proc.memory.rssMB‚ 2629.82421875 disk.rbd2.in‚0 disk.dm-1.out‚ 676.2421875$ memory_percent‚93.65031719207764* proc.memory.percent‚16.051173210144043% disk./.usageGB‚1108.3682670593262 cpu‚25.320748683366723 proc.cpu.threads‚17 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.in‚ 0.57421875 disk.rbd2.out‚ 666.353515625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875) disk./.usagePercent‚74.38352438527062 disk.rbd1.out‚4.96875 disk.sda.out‚ 676.2421875 disk.sda2.out‚0( disk.nvme0c0n1.out‚83212.10302734375# disk.dm-0.out‚83212.10302734375) proc.memory.availableMB‚ 1040.33203125 network.sent‚ 2.9824685e+07 network.recv‚ 3.605373e+06 disk.drbd1777.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚83212.10302734375 disk.rbd0.out‚0¦aöûŸþ*‚(:” Ì–æÔ§…É gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.839' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.367799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ÏíÊÿ*‚(:º Ì–æÔ±»ºÌ disk.sda1.in‚0! disk.nvme0n1.in‚ 1778.26171875 cpu‚25.320339076115054 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1778.26171875( disk.nvme0c0n1.out‚83212.40673828125$ memory_percent‚93.65098476409912* proc.memory.percent‚16.051173210144043) disk./.usagePercent‚74.38181092684394& disk.nvme0n1.out‚83212.40673828125 disk.rbd0.out‚0 disk.rbd2.out‚ 666.443359375 network.recv‚ 3.610101e+06 disk.sda.in‚ 0.57421875 disk.sda.out‚ 676.65625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.57421875 disk.rbd1.in‚0% disk./.usageGB‚1108.3427352905273 disk.rbd3.in‚0 diV«kÎ{sk.drbd1777.out‚0 disk.rbd2.in‚0 network.sent‚ 2.9830573e+07 disk.dm-1.out‚ 676.65625 disk.sda3.in‚ 0.57421875 disk.drbd1777.in‚0 disk.rbd1.out‚4.96875 disk.rbd3.out‚0 disk.sda3.out‚ 676.65625 disk.dm-0.in‚ 1778.26171875# disk.dm-0.out‚83212.40673828125) proc.memory.availableMB‚ 1040.22265625# proc.memory.rssMB‚ 2629.82421875<ïÂ4€+/ _wandbruntime‚10009  _runtime‚10009n._É+à % _timestamp‚1.7884475699225724e+09 _runtime‚10009.967224739 _step‚967 & lr-AdamW/pg1‚6.834710478575582e-07 & lr-AdamW/pg2‚6.834710478575582e-07 trainer/global_step‚11799 e5¹Î‚+È _step‚967 & lr-AdamW/pg1‚6.834710478575582e-07 & lr-AdamW/pg2‚6.834710478575582e-07 trainer/global_step‚11799 % _timestamp‚1.7884475699225724e+09 _runtime‚10009.967224739ÇÈýð™4ƒ+/ _wandbruntime‚10009  _runtime‚10009±ô$(„+žP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174169249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 394243791 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1316406224560710 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884475699233334e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174169249 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1155347946308608 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1125378085052416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 trainer/global_step‚11799 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21892374841856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29969861256192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223751 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1337440164522496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1315545163079680 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1337440164522496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194784040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220074536 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1294550139493376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21856085067334 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1337437537921536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21894975051264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚485 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21894975051264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174169255 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174169255 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174169249 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1155347946308608 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1125378085052416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 trainer/global_step‚11799 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21892374841856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚29969861256192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223751 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1337440164522496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚76627968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1315545163079680 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1337440164522496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194784040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220074536 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1294550139493376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21856085067334 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1337437537921536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21894975051264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚485 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21894975051264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81778699 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1316514395077650 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256221 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1125470501240320 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1125470473105920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1315653102785024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174183485 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194799797 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113020628 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29972408216064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174183491 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174183485 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81779166 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1294656506095616 _step‚969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21860512694878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174183491 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21896807393280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194799327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1316517018790494 _runtime‚10010.705535132 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 394276287 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220092796 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1315653102785024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1337549936569856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376Ú”3‘ˆ+­N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174183491 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174183485 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81779166 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1294656506095616 _step‚969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21860512694878 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.pea¤Â>‚¢"k‚839 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174183491 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21896807393280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194799327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1316517018790494 _runtime‚10010.705535132 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 394276287 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220092796 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1315653102785024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1337549936569856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21896807393280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113020631 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21894207170560 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220072736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220092796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚470 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 394690560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2711634944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81778699 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1316514395077650 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256221 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1125470501240320 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1125470473105920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1315653102785024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174183485 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194799797 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113020628 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚29972408216064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575É¿øyü4‰+/ _wandbruntime‚10010  _runtime‚10010´ÀÙ]Š+× .loss/FBt4/detection_loss‚2.216149278812938 0loss/FRt1/assignment_loss‚1.2845867743679122 0loss/FRt2/assignment_loss‚2.2654976147525714 0loss/FRt3/assignment_loss‚3.3330360019908234 1loss/FBt2/assignment_loss‚0.20150776864320505 1loss/SRqqt3/detection_loss‚2.1610057074576616 .loss/FBt2/detection_loss‚2.324162757396698 /loss/FBt3/detection_loss‚1.9973237216472626 0loss/FRt4/assignment_loss‚1.8671993787695722 1loss/FBt1/assignment_loss‚0.11708018314955798 _runtime‚10010.706735722 2loss/SRqqt3/assignment_loss‚1.3356687121558934 /loss/FRt1/detection_loss‚1.5854441862480313 1loss/SRqqt1/detection_loss‚0.9179181308589048 1loss/SRqqt4/detection_loss‚1.4040462426268137 .loss/FBt1/detection_loss‚0.372543668821347 % _timestamp‚1.7884475706621249e+09 epoch‚9 2loss/SRqqt4/assignment_loss‚0.9128174989197689 .loss/FRt4/detection_loss‚2.276844241633648 0loss/SRqqt2/detection_loss‚1.634342527359116 _step‚970 1loss/SRqqt1/assignment_loss‚0.792529152913226 0loss/FBt3/assignment_loss‚0.1754821539031003 0loss/FBt4/assignment_loss‚0.2714582157328146 /loss/FRt2/detection_loss‚2.0777422155974046 %loss/total_loss‚35.36963066541459 2loss/SRqqt2/assignment_loss‚1.5251471511639503 /loss/FRt3/detection_loss‚2.3200973804923706 trainer/global_step‚11799¾JŸUb‹+Ü trainer/global_step‚11799 .loss/FBt4/detection_loss‚2.216149278812938 0loss/FRt1/assignment_loss‚1.2845867743679122 0loss/FRt2/assignment_loss‚2.2654976147525714 0loss/FRt3/assignment_loss‚3.3330360019908234 1loss/FBt2/assignment_loss‚0.20150776864320505 1loss/SRqqt3/detection_loss‚2.1610057074576616 .loss/FBt2/detection_loss‚2.324162757396698 /loss/FBt3/detection_loss‚1.9973237216472626 0loss/FRt4/assignment_loss‚1.8671993787695722 1loss/FBt1/assignment_loss‚0.11708018314955798 _runtime‚10010.706735722 2loss/SRqqt3/assignment_loss‚1.3356687121558934 /loss/FRt1/detection_loss‚1.5854441862480313 1loss/SRqqt1/detection_loss‚0.9179181308589048 1loss/SRqqt4/detection_loss‚1.4040462426268137 .loss/FBt1/detection_loss‚0.372543668821347 % _timestamp‚1.7884475706621249e+09 epoch‚9 2loss/SRqqt4/assignment_loss‚0.9128174989197689 .loss/FRt4/detection_loss‚2.276844241633648 0loss/SRqqt2/detection_loss‚1.634342527359116 _step‚970 1loss/SRqqt1/assignment_loss‚0.792529152913226 0loss/FBt3/assignment_loss‚0.1754821539031003 0loss/FBt4/assignment_loss‚0.2714582157328146 /loss/FRt2/detection_loss‚2.0777422155974046 %loss/total_loss‚35.36963066541459 2loss/SRqqt2/assignment_loss‚1.5251471511639503 /loss/FRt3/detection_loss‚2.3200973804923706ÊŒct”Œ+‚(:‰ Û–æÔ©«¡É gpu.0.gpu‚94.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F¡G+‚(:¼ Û–æÔìÓ¼Ë disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.614737e+06# disk.nvme0c0n1.in‚ 1778.26171875" disk.dm-0.out‚83310.0068359375 disk.dm-1.in‚ 0.57421875 disk.sda2.out‚0# proc.memory.rssMB‚ 2629.82421875 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 2.983682e+07 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.57421875% disk.nvme0n1.out‚83310.0068359375 cpu‚25.3199157266327 disk.sda3.in‚ 0.57421875 disk.dm-1.out‚ 677.06640625# memory_percent‚93.6542272567749% disk./.usageGB‚1108.3429489135742 disk.sda1.out‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1039.69140625' disk.nvme0c0n1.out‚83310.0068359375 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚4.96875 disk.rbd2.out‚ 666.443359375! disk.nvme0n1.in‚ 1778.26171875* proc.memory.percent‚16.051173210144043) disk./.usagePercent‚74.38182526326585 disk.sda3.out‚ 677.06640625 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 677.06640625¬¤ð”Ž+‚(:‰ ê–æÔʲƒÉ gpu.0.gpu‚89.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æl:iP+‚(:Å ê–æÔßï•Ë disk.drbd1777.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1039.91796875# proc.memory.rssMB‚ 2629.82421875* proc.memory.percent‚16.051173210144043 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.in‚ 0.57421875$ memory_percent‚93.65284442901611# disk.nvme0c0n1.in‚ 1778.26171875( disk.nvme0c0n1.out‚83310.87548828125 network.recv‚ 3.620951e+06! disk.rbd2.out‚666.54541015625& disk.nvme0n1.out‚83310.87548828125 disk.rbd0.in‚0# disk.dm-0.out‚83310.87548828125 cpu‚25.319514948118684! disk.nvme0n1.in‚ 1778.26171875 disk.sda2.out‚0 disk.sda3.out‚ 677.51171875) disk./.usagePercent‚74.38182679931107 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 677.51171875 disk.rbd1.out‚4.96875 disk.rbd3.out‚0 disk.sda3.in‚ 0.57421875 disk.dm-0.in‚ 1778.26171875 network.sent‚ 2.9945184e+07 disk.drbd1777.out‚0 disk.sda.out‚ 677.51171875 disk.sda1.out‚0 disk.rbd3.in‚0% disk./.usageGB‚1108.3429718017578 disk.dm-1.in‚ 0.57421875Ë &Ù4+/ _wandbruntime‚10048  _runtime‚10048†YòuÇ‘+Á trainer/global_step‚11849 % _timestamp‚1.7884476088886125e+09 _runtime‚10048.933279156 _step‚971 % lr-AdamW/pg1‚4.68865726863479e-07 % lr-AdamW/pg2‚4.68865726863479e-07` uaÌ’+Æ % _timestamp‚1.7884476088886125e+09 _runtime‚10048.933279156 _step‚971 % lr-AdamW/pg1‚4.68865726863479e-07 % lr-AdamW/pg2‚4.68865726863479e-07 trainer/global_step‚11849Ë?\4“+/ _wandbruntime‚10048  _runtime‚10048Yö[H$(”+žP VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1342926154037760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82107194 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1342928780638720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113471235 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1129997515862016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220969554 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21986605900288 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _runtime‚10048.934341957 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850603 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1342928780638720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1320942174738432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174881055 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220989588 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1160091979888640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174881049 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21986605900288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1160091894872064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1342926154037760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220989588 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320942174738432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21984005690880 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195578949 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195578426 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30094464026624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320942148346880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113471232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884476088894296e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1299867223634944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚56882176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376®æ0+•+£P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1129997515862016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220969554 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21986605900288 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 _runtime‚10048.934341957 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395850603 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1342928780638720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1320942174738432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174881055 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220989588 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1160091979888640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174881049 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21986605900288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1160091894872064 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1342926154037760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220989588 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320942174738432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21984005690880 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195578949 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195578426 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30094464026624 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚2649 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320942148346880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113471232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 % _timestamp‚1.7884476088894296e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1299867223634944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚56882176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1342926154037760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82107194 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1342928780638720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113471235Ìj›îà4–+/ _wandbruntime‚10049  _runtime‚10049jÜòÚ.'—+¨N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195594857 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1343038552686080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚30096877071360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174895285 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174895285 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21985838019584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1299973564809216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321050088052224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 414947328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1130089875781120 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321050114443776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚56868864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82113928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚973 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195594356 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21949395134610 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1160186837855744TÑX 3'˜+­N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195594857 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1343038552686080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚30096877071360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174895285 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174895285 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21985838019584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1299973564809216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321050088052224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 414947328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1130089875781120 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321050114443776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚56868864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82113928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚973 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195594356 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21949395134610 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1160186837855744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233Íȇ4™+/ _wandbruntime‚10049  _runtime‚10049ø;yL\š+Ö 1loss/FBt1/assignment_loss‚0.05256567486149119 /loss/FBt3/detection_loss‚2.2479763329029083 _step‚974 0loss/SRqqt2/detection_loss‚1.522016758909044 1loss/SRqqt1/detection_loss‚0.6481811321470179 1loss/SRqqt3/detection_loss‚2.1711248101726657 /loss/FRt3/assignment_loss‚2.957675261930986 2loss/SRqqt1/assignment_loss‚0.6621432684438747 0loss/FBt2/assignment_loss‚0.4431079423666233 /loss/FRt1/detection_loss‚1.8364826465982038 epoch‚9 $ _timestamp‚1.788447609843321e+09 1loss/SRqqt3/assignment_loss‚1.437188753918294 /loss/FRt4/detection_loss‚2.1152024587919547 /loss/FBt2/detection_loss‚3.0577256083488464 _runtime‚10049.887921602 1loss/SRqqt2/assignment_loss‚1.279710335009124 /loss/FRt3/detection_loss‚2.7033346279651704 %loss/total_loss‚35.94324472327787 0loss/FRt1/assignment_loss‚0.8768420364405658 2loss/SRqqt4/assignment_loss‚0.8740801004549632 1loss/FBt4/assignment_loss‚0.18085952618760834 .loss/FRt2/detection_loss‚2.564785096575232 /loss/FBt1/detection_loss‚0.4680256952709852 /loss/FBt4/detection_loss‚1.7156047175327938 /loss/FRt4/assignment_loss‚1.723877648974574 1loss/FBt3/assignment_loss‚0.22915582850109786 0loss/SRqqt4/detection_loss‚1.834042727947235 trainer/global_step‚11849 0loss/FRt2/assignment_loss‚2.3415357330266167¶Ÿa›+Û 1loss/SRqqt1/detection_loss‚0.6481811321470179 1loss/SRqqt3/detection_loss‚2.1711248101726657 /loss/FRt3/assignment_loss‚2.957675261930986 2loss/SRqqt1/assignment_loss‚0.6621432684438747 0loss/FBt2/assignment_loss‚0.4431079423666233 /loss/FRt1/detection_loss‚1.8364826465982038 epoch‚9 $ _timestamp‚1.788447609843321e+09 1loss/SRqqt3/assignment_loss‚1.437188753918294 /loss/FRt4/detection_loss‚2.1152024587919547 /loss/FBt2/detection_loss‚3.0577256083488464 _runtime‚10049.887921602 1loss/SRqqt2/assignment_loss‚1.279710335009124 /loss/FRt3/detection_loss‚2.7033346279651704 %loss/total_loss‚35.94324472327787 0loss/FRt1/assignment_loss‚0.8768420364405658 2loss/SRqqt4/assignment_loss‚0.8740801004549632 1loss/FBt4/assignment_loss‚0.18085952618760834 .loss/FRt2/detection_loss‚2.564785096575232 /loss/FBt1/detection_loss‚0.4680256952709852 /loss/FBt4/detection_loss‚1.7156047175327938 /loss/FRt4/assignment_loss‚1.723877648974574 1loss/FBt3/assignment_loss‚0.22915582850109786 0loss/SRqqt4/detection_loss‚1.834042727947235 trainer/global_step‚11849 0loss/FRt2/assignment_loss‚2.3415357330266167 1loss/FBt1/assignment_loss‚0.05256567486149119 /loss/FBt3/detection_loss‚2.2479763329029083 _step‚974 0loss/SRqqt2/detection_loss‚1.522016758909044ÎUé«Í“œ+‚(:ˆ ù–æÔ·³†É gpu.0.gpu‚88.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚37.0 gpu.0.powerWatts‚90.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00³³M+‚(: ù–æÔ¸Ë! disk.nvme0n1.in‚ 1778.26171875 cpu‚25.31913827746418 disk.rbd3.in‚0& disk.nvme0n1.out‚83408.48681640625 disk.rbd0.out‚0 network.recv‚ 3.625419e+06) disk./.usagePercent‚74.38183627158983 disk.dm-1.in‚ 0.57421875 disk.sda2.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 678.19921875( proc.memory.availableMB‚ 1039.4609375 proc.cpu.threads‚17% disk./.usageGB‚1108.3431129455566 disk.dm-0.in‚ 1778.26171875 disk.drbd1777.in‚0 disk.sda.out‚ 678.19921875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚93.6556339263916 disk.rbd1.out‚4.96875( disk.nvme0c0n1.out‚83408.48681640625* proc.memory.percent‚16.051173210144043# disk.dm-0.out‚83408.48681640625 disk.sda.in‚ 0.57421875 disk.sda3.in‚ 0.57421875# proc.memory.rssMB‚ 2629.82421875 disk.rbd1.in‚0! disk.rbd2.out‚666.54541015625 network.sent‚ 2.9951154e+07 disk.dm-1.out‚ 678.19921875 disk.rbd3.out‚0ôOºð”ž+‚(:‰ ˆ—æÔ’æ²É gpu.0.gpu‚57.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:•IŸ+‚(:¾ ˆ—æÔ¢¸êÊ disk.drbd1777.out‚0) disk./.usagePercent‚74.38183627158983 disk.rbd1.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 1039.796875 disk.rbd1.out‚4.96875# disk.nvme0c0n1.in‚ 1778.26171875" disk.dm-0.out‚83408.9267578125! disk.rbd2.out‚668.19775390625 disk.sda2.in‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2629.82421875 network.sent‚ 3.006016e+07 disk.sda.in‚ 0.57421875 disk.sda.out‚ 678.81640625 disk.rbd3.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚83408.9267578125 disk.drbd1777.in‚0$ memory_percent‚93.65358352661133 disk.rbd2.in‚0 disk.dm-0.in‚ 1778.26171875 disk.dm-1.out‚ 678.81640625* proc.memory.percent‚16.051173210144043 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.31874028680559 proc.cpu.threads‚17 disk.dm-1.in‚ 0.57421875 network.recv‚ 3.631937e+06% disk./.usageGB‚1108.3431129455566 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.26171875% disk.nvme0n1.out‚83408.9267578125 disk.sda3.out‚ 678.81640625;øýД +‚(:‰ ——æÔÜ gpu.0.gpu‚71.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚233.284' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.6568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øb§eH¡+‚(:½ ——æÔöô¡Ë disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875$ memory_percent‚93.65205764770508 network.sent‚ 3.0066513e+07 network.recv‚ 3.636683e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 1778.26171875% disk.nvme0n1.out‚83506.5908203125 disk.sda3.out‚ 679.55859375 cpu‚25.3183144347364 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚74.38183371151449! disk.rbd2.out‚668.19775390625 disk.sda.in‚ 0.57421875 disk.sda2.in‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.out‚0' proc.memory.availableMB‚ 1040.046875 disk.sda1.in‚0$ disk./.usageGB‚1108.343074798584 disk.rbd1.out‚4.96875* proc.memory.percent‚16.051173210144043! disk.nvme0n1.in‚ 1778.26171875' disk.nvme0c0n1.out‚83506.5908203125# proc.memory.rssMB‚ 2629.82421875 proc.cpu.threads‚17 disk.sda.out‚ 679.55859375" disk.dm-0.out‚83506.5908203125 disk.dm-1.out‚ 679.55859375­aá4¢+/ _wandbruntime‚10088  _runtime‚10088ÕnHÊ£+Ä ' lr-AdamW/pg1‚2.9454137314498646e-07 ' lr-AdamW/pg2‚2.9454137314498646e-07 trainer/global_step‚11899 $ _timestamp‚1.788447648110555e+09 _runtime‚10088.155199018 _step‚975”F´ Ϥ+É ' lr-AdamW/pg1‚2.9454137314498646e-07 ' lr-AdamW/pg2‚2.9454137314498646e-07 trainer/global_step‚11899 $ _timestamp‚1.788447648110555e+09 _runtime‚10088.155199018 _step‚975Ï—ÂJú4¥+/ _wandbruntime‚10088  _runtime‚10088ûÏ{rd¦+P `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30219039165440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1134616890402816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1164835929568256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1348417396754944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 ^IDeviceStatsMonitor.on_train_batch_EÍ©C¿!start/active_bytes.large_pool.allocated‚1326339160005632 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚78725120 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1327223303437126 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1305184256920576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22078236749312 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1164836008293376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175592855 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1134616918537216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1326339133614080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175592849 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1305184282348544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221884606 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1327225927149762 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22039046516550 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1348414770153984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175592855 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82444187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221884606 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1348414770153984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22078236749312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175592849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82443713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221904640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196374745 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113931032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113931035 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚11899 $ _timestamp‚1.788447648111267e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196375222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397477455 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _step‚976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1348417396754944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30219089756160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1348414770153984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22078236749312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175592849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82443713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221904640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196374745 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113931032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113931035 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 trainer/global_step‚11899 $ _timestamp‚1.788447648111267e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196375222 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397477455 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 _step‚976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1348417396754944 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚477 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30219089756160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1348527168802304 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221902840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1327331473954066 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1134709306590720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22080069091328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22080069091328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1134709278456320 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1305290598094848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚11899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1326447099710976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22077468868608 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221902840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221922900 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚977 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175607085 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82451026 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1164930815902208 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22040875859218 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175607085 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788447648859019e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1326447073319424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175607091 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196391257 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196390792 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397509925 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1326447099710976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30221585926144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1348524542188032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221922900 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1305290623522816úE)z•ª+¬N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1305290598094848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚11899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚48480256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1326447099710976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22077468868608 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221902840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221922900 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚977 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 2598284876 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175607085 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82451026 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1164930815902208 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_poox³É†l.freed‚22040875859218 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175607085 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788447648859019e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1326447073319424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175607091 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196391257 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196390792 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397509925 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚76614656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1326447099710976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30221585926144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1348524542188032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221922900 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1305290623522816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 _runtime‚10088.904093146 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175607091 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82450564 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1348527168802304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1348527168802304 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221902840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1327331473954066 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1134709306590720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22080069091328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22080069091328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1134709278456320ÑãbÌ4«+/ _wandbruntime‚10088  _runtime‚10088o4Á^¬+Ø /loss/FRt2/detection_loss‚2.2159127325427774 /loss/FRt3/detection_loss‚2.6850589679622012 1loss/SRqqt2/detection_loss‚1.6267476216250776 .loss/FBt1/detection_loss‚0.590098051345334 %loss/total_loss‚37.71718473134523 1loss/SRqqt1/detection_loss‚0.8842504029162228 _runtime‚10088.905533548 0loss/FRt3/assignment_loss‚3.1126250205216586 1loss/FBt2/assignment_loss‚0.23663215837586904 0loss/FBt4/assignment_loss‚0.7366416796252856 -loss/FBt3/detection_loss‚4.65080714225769 epoch‚9 % _timestamp‚1.7884476488606713e+09 1loss/SRqqt3/detection_loss‚2.1721879603231655 .loss/FBt2/detection_loss‚2.313874252140522 0loss/FRt1/assignment_loss‚0.8867195689678192 0loss/FRt2/assignment_loss‚1.9615437461405385 2loss/SRqqt2/assignment_loss‚1.0729018820851457 /loss/FRt1/detection_loss‚1.5763917805813252 .loss/FRt4/detection_loss‚2.265351722186262 0loss/SRqqt4/detection_loss‚1.415893661617129 trainer/global_step‚11899 3loss/FBt3/assignment_loss‚0.0006900938169565052 2loss/SRqqt1/assignment_loss‚0.8034871644267696 .loss/FBt4/detection_loss‚2.111501097679138 _step‚978 0loss/FRt4/assignment_loss‚1.9755436385219747 2loss/SRqqt3/assignment_loss‚1.1411998486694168 2loss/SRqqt4/assignment_loss‚1.1190193760622706 0loss/FBt1/assignment_loss‚0.1621051609546788q«2Ñc­+Ý trainer/global_step‚11899 3loss/FBt3/assignment_loss‚0.0006900938169565052 2loss/SRqqt1/assignment_loss‚0.8034871644267696 .loss/FBt4/detection_loss‚2.111501097679138 _step‚978 0loss/FRt4/assignment_loss‚1.9755436385219747 2loss/SRqqt3/assignment_loss‚1.1411998486694168 2loss/SRqqt4/assignment_loss‚1.1190193760622706 0loss/FBt1/assignment_loss‚0.1621051609546788 /loss/FRt2/detection_loss‚2.2159127325427774 /loss/FRt3/detection_loss‚2.6850589679622012 1loss/SRqqt2/detection_loss‚1.6267476216250776 .loss/FBt1/detection_loss‚0.590098051345334 %loss/total_loss‚37.71718473134523 1loss/SRqqt1/detection_loss‚0.8842504029162228 _runtime‚10088.905533548 0loss/FRt3/assignment_loss‚3.1126250205216586 1loss/FBt2/assignment_loss‚0.23663215837586904 0loss/FBt4/assignment_loss‚0.7366416796252856 -loss/FBt3/detection_loss‚4.65080714225769 epoch‚9 % _timestamp‚1.7884476488606713e+09 1loss/SRqqt3/detection_loss‚2.1721879603231655 .loss/FBt2/detection_loss‚2.313874252140522 0loss/FRt1/assignment_loss‚0.8867195689678192 0loss/FRt2/assignment_loss‚1.9615437461405385 2loss/SRqqt2/assignment_loss‚1.0729018820851457 /loss/FRt1/detection_loss‚1.5763917805813252 .loss/FRt4/detection_loss‚2.265351722186262 0loss/SRqqt4/detection_loss‚1.415893661617129ÒO ¾Ø“®+‚(:ˆ ¦—æÔ•¥‹É gpu.0.gpu‚70.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚250.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¨Žò?¯+‚(:´ ¦—æÔàÌöÊ disk.sda.in‚ 0.57421875 disk.sda.out‚679.875 disk.sda1.out‚0& disk.nvme0n1.out‚83507.06787109375( disk.nvme0c0n1.out‚83507.06787109375 disk.rbd3.in‚0 disk.dm-0.in‚ 1778.26171875) proc.memory.availableMB‚ 1039.80859375 disk.rbd2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1778.26171875 disk.rbd0.out‚0 cpu‚25.317918578843408) disk./.usagePercent‚74.38183371151449 disk.rbd3.out‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.drbd1777.out‚0# memory_percent‚93.6535120010376 network.sent‚ 3.0071779e+07 disk.dm-1.out‚679.875 disk.rbd1.out‚4.96875# proc.memory.rssMB‚ 2629.82421875 disk.sda3.out‚679.875 proc.cpu.threads‚17 network.recv‚ 3.640482e+06$ disk./.usageGB‚1108.343074798584! disk.rbd2.out‚668.27587890625 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚83507.06787109375 disk.drbd1777.in‚0* proc.memory.percent‚16.051173210144043 disk.rbd1.in‚0’rÞ°+‚(:“ µ—æÔݘÉ gpu.0.gpu‚81.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘”.IJ±+‚(:¿ µ—æÔ¬˜Ë$ disk./.usageGB‚1108.343074798584 disk.rbd3.out‚0 disk.sda3.in‚ 0.57421875 disk.dm-0.in‚ 1778.26171875 disk.rbd1.in‚0 disk.rbd2.in‚0 network.recv‚ 3.646911e+06 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 5.0546875* proc.memory.percent‚16.051173210144043 disk.sda.in‚ 0.57421875 disk.sda2.in‚0$ memory_percent‚93.65482330322266 network.sent‚ 3.0180841e+07 disk.sda1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚83604.6650390625 proc.cpu.threads‚17 disk.sda.out‚ 680.48046875 disk.rbd3.in‚0 disk.sda3.out‚ 680.48046875 disk.rbd0.in‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2629.82421875 disk.rbd0.out‚0! disk.rbd2.out‚668.27587890625! disk.nvme0n1.in‚ 1778.26171875% disk.nvme0n1.out‚83604.6650390625" disk.dm-0.out‚83604.6650390625 disk.dm-1.out‚ 680.48046875 cpu‚25.31749591470542# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875& proc.memory.availableMB‚ 1039.59375) disk./.usagePercent‚74.38183371151449sô9”²+‚(:‰ Ä—æÔÜžêÈ gpu.0.gpu‚73.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³'®#P³+‚(:Å Ä—æÔ¬ÌàË disk.drbd1777.in‚0 disk.rbd3.out‚0 network.recv‚ 3.651269e+06 disk.rbd2.out‚668.3779296875$ memory_percent‚93.65475177764893) proc.memory.availableMB‚ 1039.60546875 disk.dm-0.in‚ 1778.26171875 disk.sda.in‚ 0.57421875 disk.sda.out‚ 680.80078125 network.sent‚ 3.0186643e+07) disk./.usagePercent‚74.38185009599668( disk.nvme0c0n1.out‚83605.49072265625# disk.dm-0.out‚83605.49072265625 disk.dm-1.in‚ 0.57421875 disk.sda3.out‚ 680.80078125 disk.dm-1.out‚ 680.80078125 disk.rbd1.out‚ 5.1171875& disk.nvme0n1.out‚83605.49072265625$ disk./.usageGB‚1108.343318939209 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.051173210144043 cpu‚25.317122605592704 disk.sda2.out‚0 disk.sda3.in‚ 0.57421875 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2629.82421875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1778.26171875# disk.nvme0c0n1.in‚ 1778.26171875©%g4´+/ _wandbruntime‚10127  _runtime‚10127+AõlƵ+À % lr-AdamW/pg1‚1.60534315636164e-07 % lr-AdamW/pg2‚1.60534315636164e-07 trainer/global_step‚11949 $ _timestamp‚1.788447687576758e+09 _runtime‚10127.621386546 _step‚979Or¤/˶+Å % lr-AdamW/pg1‚1.60534315636164e-07 % lr-AdamW/pg2‚1.60534315636164e-07 trainer/global_step‚11949 $ _timestamp‚1.788447687576758e+09 _runtime‚10127.621386546 _step‚979Ó]7U‹4·+/ _wandbruntime‚10127  _runtime‚10127ŒývS$(¸+žP O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222799658 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176304649 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚30343646887936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20034 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353903386270208 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1331736118881280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1139236293078016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22167267388928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222819692 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222819692 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 399104307 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1169579939965952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22169867598336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176304655 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197171223 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176304649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1332631842875334 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884476875774686e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1331736145272832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 _runtime‚10127.622275489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1353906012871168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22130527241158 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22169867598336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30343699575808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1331736145272832 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197171696 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1139236321212416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 _step‚980 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114390832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82780861 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399124347 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 trainer/global_step‚11949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1331736118881280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 trainer/global_step‚11949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1331736118881280 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353903386270208 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1331736118881280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2753 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1139236293078016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22167267388928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222819692 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222819692 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 399104307 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1169579939965952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22169867598336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 2600209408 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176304655 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197171223 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚80822272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176304649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1332631842875334 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884476875774686e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1331736145272832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 _runtime‚10127.622275489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1353906012871168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22130527241158 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22169867598336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚30343699575808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1331736145272832 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197171696 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1139236321212416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649Ôbšä4º+/ _wandbruntime‚10128  _runtime‚10128µ'¾Õ.'»+¨N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114400028 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222837952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82787602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1310607656808448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136777 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222817892 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222817892 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _step‚981 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22171699940352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197187171 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1354015784918528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1354015784918528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82787143 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1139328709265920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22169099717632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1139328681131520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 399136777 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22134954868702 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1354013158304256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176318891 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚11949 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176318885 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222837952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222837952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82787602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1310607656808448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136777 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222817892 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222817892 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 _step‚981 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22171699940352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197187171 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1354015784918528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1354015784918528 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82787143 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1139328709265920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22169099717632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1139328681131520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 399136777 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22134954868702 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1354013158304256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176318891 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚11949 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176318885 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222837952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1359392002386432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83116687 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1143855723887616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22224606250434 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1143855695753216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 177016449 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak¯qîÍ‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114850635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83117152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 trainer/global_step‚11999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1359392002386432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223714710 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1337133104148480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1315818399775744 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1315818374347776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22222007965766 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22258898237952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197967319 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1337133130540032 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1174324611231744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1337133130540032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1359392002386432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83116687 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1143855723887616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚50590720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22224606250434 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1143855695753216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 177016449 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114850635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83117152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 trainer/global_step‚11999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1359392002386432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223714710 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1337133104148480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1315818399775744 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1315818374347776 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22222007965766 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22258898237952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197967319 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 20788341248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚1575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1337133130540032 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1174324611231744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1337133130540032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197983196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223753004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚11999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177030691 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114859828 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22263330789376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197983647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1315924715522048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1174419582374912 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177030691 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223753004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763629 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22226435593310 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1174419503663104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1143948111941120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177030685 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22260730566656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 % _timestamp‚1.7884477274973984e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22263330789376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1338151176543326 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1359504401034752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223732944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1315924740950016 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1337241043853824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22260730566656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1337241043853824›æH%†Ì+­N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1359504401034752 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114859831 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1338148552830482 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 _step‚985 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1174419582374912 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177030691 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223753004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚50577408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763629 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22226435593310 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1174419503663104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1143948111941120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177030685 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22260730566656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 2623712844 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚78711808 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 % _timestamp‚1.7884477274973984e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22263330789376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1338151176543326 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1359504401034752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223732944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1315924740950016 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1337241043853824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22260730566656 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1337241043853824 TEDeviceStatsMonitoý¿”Û­r.on_train_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83123816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 414947328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177030685 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1337241070245376 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197983196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223753004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 trainer/global_step‚11999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177030691 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚839 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114859828 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22263330789376 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197983647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚2649 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1315924715522048 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1364880618502656 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1343448921751750 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _step‚988 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1321135433061376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚10206.248011135 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22350529086976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198763555 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22350529086976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83452615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224649796 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22313488690374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 177728249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177728249 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1364883245103616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1148475126562816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1342530089415680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83453120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22316086975042 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402358011 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115310435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1343451545464386 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1321135458489344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1364880618502656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224629762 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1364883245103616 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1342530115807232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22353129296384 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1342530089415680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24124 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1364880618502656 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1343448921751750 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 20424465233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 3303014400 _step‚988 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1321135433061376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚10206.248011135 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 2600209408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1074 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1074 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22350529086976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20034 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198763555 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22350529086976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1575 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83452615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚28134400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224649796 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚2649 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22313488690374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1074 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 177728249 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 18093162496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177728249 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1364883245103616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 23672651776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 2626600960 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1148475126562816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24124 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2730509312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 23672651776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚2649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚839 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 3245630568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚2649 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1342530089415680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83453120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 2623712636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2338332672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 17729555456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22316086975042 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 3303014400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402358011 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2753 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 3247569920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115310435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚52687872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1343451545464386 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1321135458489344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 20369637376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1364880618502656 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 20788341248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 23672651776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚1575 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 20369637376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224629762 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚609 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1364883245103616 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1342530115807232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 2598284668 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 2626600960 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22353129296384 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1342530089415680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 18093162496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198778873 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatŸ¾av%sMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22354961638400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224647996 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115319628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22315318033042 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _step‚989 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83459245 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚485 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10206.988331211 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1343557092268690 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22352361415680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198779358 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22352361415680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1342638055512576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177742485 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224668056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30596122194432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788447766943212e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402410547 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1364993017150976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22354961638400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1364993017150976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22317916317918 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177742485 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1148567514616320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1342638055512576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1342638029121024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 414947328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224668056 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1321241774235648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390481 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1342638029121024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1343559715981534 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1364990390536704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177742491 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1179163556001792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568ã.GC2'Þ+¬N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22315318033042 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1575 _step‚989 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83459245 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚485 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10206.988331211 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1343557092268690 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22352361415680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198779358 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22352361415680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1342638055512576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177742485 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 23672651776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224668056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚30596122194432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788447766943212e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402410547 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20060 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚2649 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1074 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1364993017150976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22354961638400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 2626614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2753 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1364993017150976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22317916317918 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177742485 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 2626614272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2730509312 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1148567514616320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 2600222720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚52674560 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1342638055512576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1342638029121024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 20788341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24124 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 414947328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 23672651776 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 20369637376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224668056 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 20424465233 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1321241774235648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2649 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402390481 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1342638029121024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚28134400 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1343559715981534 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1074 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1364990390536704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177742491 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚80808960 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1179163556001792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3245630568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 3303014400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 3247569920 trainer/global_step‚12049 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198778873 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 20788341248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22354961638400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 17729555456 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224647996 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 18093162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 2600222720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 23672651776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚2649 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1575 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚1575 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115319628Ýô/74ß+/ _wandbruntime‚10206  _runtime‚10206¼ F6Zà+Ô 0loss/FRt2/assignment_loss‚1.8245741572490959 /loss/FRt3/assignment_loss‚3.421577211469412 2loss/SRqqt2/assignment_loss‚0.8412912469458483 1loss/SRqqt4/assignment_loss‚1.106410635148429 .loss/FRt1/detection_loss‚1.347672960953787 1loss/SRqqt1/detection_loss‚0.9211798446466688 0loss/SRqqt3/detection_loss‚2.320931438137503 /loss/FBt3/detection_loss‚2.2619603514671325 &loss/total_loss‚38.916786956705984 1loss/FBt4/assignment_loss‚0.07918269086318712 trainer/global_step‚12049 _step‚990 /loss/FRt1/assignment_loss‚1.019453617433707 2loss/SRqqt1/assignment_loss‚0.6821757917553396 .loss/FBt4/detection_loss‚2.077494959036509 /loss/FRt4/assignment_loss‚2.599824863044839 2loss/SRqqt3/assignment_loss‚1.5397141330382402 0loss/FBt1/assignment_loss‚0.0697125130124004 /loss/FRt2/detection_loss‚2.2208812153616617 1loss/SRqqt4/detection_loss‚1.9621410588310524 /loss/FBt1/detection_loss‚0.5949338709278141 .loss/FBt2/detection_loss‚4.141782283782959 1loss/FBt3/assignment_loss‚0.01297363507328555 % _timestamp‚1.7884477669448614e+09 /loss/FBt2/assignment_loss‚1.193872094127073 .loss/FRt3/detection_loss‚2.266182278333872 /loss/FRt4/detection_loss‚2.6271965488007196 1loss/SRqqt2/detection_loss‚1.7836675572654475 epoch‚9 _runtime‚10206.989424199¦w>y_á+Ù /loss/FRt4/assignment_loss‚2.599824863044839 2loss/SRqqt3/assignment_loss‚1.5397141330382402 0loss/FBt1/assignment_loss‚0.0697125130124004 /loss/FRt2/detection_loss‚2.2208812153616617 1loss/SRqqt4/detection_loss‚1.9621410588310524 /loss/FBt1/detection_loss‚0.5949338709278141 .loss/FBt2/detection_loss‚4.141782283782959 1loss/FBt3/assignment_loss‚0.01297363507328555 % _timestamp‚1.7884477669448614e+09 /loss/FBt2/assignment_loss‚1.193872094127073 .loss/FRt3/detection_loss‚2.266182278333872 /loss/FRt4/detection_loss‚2.6271965488007196 1loss/SRqqt2/detection_loss‚1.7836675572654475 epoch‚9 _runtime‚10206.989424199 0loss/FRt2/assignment_loss‚1.8245741572490959 /loss/FRt3/assignment_loss‚3.421577211469412 2loss/SRqqt2/assignment_loss‚0.8412912469458483 1loss/SRqqt4/assignment_loss‚1.106410635148429 .loss/FRt1/detection_loss‚1.347672960953787 1loss/SRqqt1/detection_loss‚0.9211798446466688 0loss/SRqqt3/detection_loss‚2.320931438137503 /loss/FBt3/detection_loss‚2.2619603514671325 &loss/total_loss‚38.916786956705984 1loss/FBt4/assignment_loss‚0.07918269086318712 trainer/global_step‚12049 _step‚990 /loss/FRt1/assignment_loss‚1.019453617433707 2loss/SRqqt1/assignment_loss‚0.6821757917553396 .loss/FBt4/detection_loss‚2.077494959036509ÞXë€êŸâ+‚(:” ž˜æÔêÉ gpu.0.gpu‚94.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚150.734' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.146800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñÎgžJã+‚(:¿ ž˜æÔ²‘ÏË# disk.dm-0.out‚83900.72412109375$ memory_percent‚93.65293979644775 disk.sda1.in‚0 disk.rbd3.out‚0 network.recv‚ 3.680081e+06) disk./.usagePercent‚74.38183550356723 disk.sda.in‚ 0.57421875 network.sent‚ 3.0427066e+07 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 684.09375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1778.26171875* proc.memory.percent‚16.055727005004883 cpu‚25.314751517617744 disk.drbd1777.out‚0 disk.rbd1.out‚ 5.275390625! disk.nvme0n1.in‚ 1778.26171875! disk.rbd2.out‚670.39794921875& disk.nvme0n1.out‚83900.72412109375) proc.memory.availableMB‚ 1039.90234375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚ 0.57421875 disk.dm-0.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.dm-1.out‚ 684.09375" proc.memory.rssMB‚ 2630.5703125 disk.sda2.out‚0 disk.rbd0.in‚0% disk./.usageGB‚1108.3431015014648 disk.drbd1777.in‚0 disk.sda.out‚ 684.09375 disk.sda1.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚83900.72412109375WŠÁŸä+‚(:” ­˜æﴀɃ gpu.0.gpu‚95.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ÚÄJå+‚(:¿ ­˜æÔÆ£ÑÌ disk.sda.out‚ 684.609375 disk.rbd2.in‚0* proc.memory.percent‚16.055727005004883 cpu‚25.3143591638751 disk.sda2.in‚0# disk.dm-0.out‚83998.16943359375 disk.rbd1.out‚ 5.275390625( proc.memory.availableMB‚ 1039.4140625 disk.drbd1777.out‚0% disk./.usageGB‚1108.3431015014648! disk.nvme0n1.in‚ 1778.26171875! disk.rbd2.out‚671.90966796875$ memory_percent‚93.65592002868652" proc.memory.rssMB‚ 2630.5703125 proc.cpu.threads‚17 disk.sda3.in‚ 0.57421875 disk.sda3.out‚ 684.609375# disk.nvme0c0n1.in‚ 1778.26171875 disk.dm-1.out‚ 684.609375 network.recv‚ 3.686362e+06 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚83998.16943359375& disk.nvme0n1.out‚83998.16943359375) disk./.usagePercent‚74.38183550356723 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 network.sent‚ 3.0536051e+07 disk.sda.in‚ 0.57421875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1778.26171875 disk.dm-1.in‚ 0.57421875 disk.rbd1.in‚0žpNvžæ+‚(:“ ¼˜æÔ­ÚœÉ gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚40.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÌ6,Qç+‚(:Æ ¼˜æÔÅ—…Ë disk.rbd3.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.57421875 disk.drbd1777.in‚0 network.recv‚ 3.69065e+06 disk.sda3.out‚ 685.03515625 disk.dm-1.out‚ 685.03515625 disk.drbd1777.out‚0) disk./.usagePercent‚74.38183550356723 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1778.26171875# disk.dm-0.out‚83998.46826171875$ memory_percent‚93.74394416809082# proc.memory.rssMB‚ 2645.33203125 disk.sda2.out‚0 disk.dm-0.in‚ 1778.26171875 disk.rbd1.out‚ 5.275390625* proc.memory.percent‚16.145825386047363 disk.sda2.in‚0 disk.rbd1.in‚0! disk.rbd2.out‚671.98779296875 network.sent‚ 3.0542148e+07 disk.rbd0.out‚0( disk.nvme0c0n1.out‚83998.46826171875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1024.9921875 proc.cpu.threads‚17% disk./.usageGB‚1108.3431015014648& disk.nvme0n1.out‚83998.46826171875 disk.rbd2.in‚0 cpu‚25.31668280216518! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.in‚ 0.57421875 disk.sda.in‚ 0.57421875 disk.sda.out‚ 685.03515625ò‚$É‘è+‚(:† ˘æÔä¯õÈ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚34.0 gpu.0.powerWatts‚87.376' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.4752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zó¶]Qé+‚(:Æ ˘æÔµ¢¡Ë! disk.nvme0n1.in‚ 1778.26171875 disk.sda3.in‚ 0.57421875 disk.dm-1.out‚ 685.50390625$ memory_percent‚94.18535232543945% disk./.usageGB‚1108.3443870544434 disk.drbd1777.in‚0 disk.sda.out‚ 685.50390625 disk.dm-1.in‚ 0.57421875 disk.rbd1.out‚ 5.275390625 disk.rbd2.in‚0 disk.sda.in‚ 0.57421875 disk.sda1.in‚0 disk.sda3.out‚ 685.50390625 disk.dm-0.in‚ 1778.26171875 network.sent‚ 3.0548299e+07 network.recv‚ 3.695284e+06& disk.nvme0n1.out‚84097.41064453125( disk.nvme0c0n1.out‚84097.41064453125) disk./.usagePercent‚74.38192177810626 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1778.26171875* proc.memory.percent‚16.586947441101074 proc.cpu.threads‚17 disk.drbd1777.out‚0! disk.rbd2.out‚671.98779296875 disk.sda2.out‚0 disk.sda1.out‚0# disk.dm-0.out‚84097.41064453125 cpu‚25.323364912353078 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 952.671875# proc.memory.rssMB‚ 2717.60546875¤Þ1ª“ê+‚(:ˆ Ú˜æÔ†ÂÎÉ gpu.0.gpu‚91.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚36.0 gpu.0.powerWatts‚89.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï1Ú–Oë+‚(:Ä Ú˜æÔûÕûÌ disk.dm-1.in‚ 0.57421875* proc.memory.percent‚16.586947441101074) disk./.usagePercent‚74.38192177810626 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 672.0859375 disk.sda2.in‚0 disk.sda3.out‚ 685.83203125( proc.memory.availableMB‚ 952.62890625 proc.cpu.threads‚17 network.sent‚ 3.0553646e+07 disk.sda.in‚ 0.57421875 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 1778.26171875 cpu‚25.330019882439707 network.recv‚ 3.699117e+06% disk./.usageGB‚1108.3443870544434 disk.sda.out‚ 685.83203125 disk.sda2.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚84098.11962890625# disk.dm-0.out‚84098.11962890625 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 1778.26171875 disk.dm-0.in‚ 1778.26171875# proc.memory.rssMB‚ 2717.60546875 disk.drbd1777.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚84098.11962890625 disk.dm-1.out‚ 685.83203125 disk.rbd0.out‚0$ memory_percent‚94.18561458587646 disk.drbd1777.in‚0 disk.rbd1.out‚ 5.275390625 disk.sda3.in‚ 0.57421875²²ôn4ì+/ _wandbruntime‚10288  _runtime‚10288je€; í+†U [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1373019353791488 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 199998640 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 $ _timestamp‚1.788447848946451e+09 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788341248 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1373021328804352 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚83922468 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1575 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚22451514270720 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1373021328804352 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚22449563552256 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚30719218876928 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2649 _step‚991 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 178845737 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 199998507 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 225752220 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.L›|"Q allocated‚1575 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚22449563552256 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581978 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚22412327267202 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1575 _runtime‚10288.991510766 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 404597957 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 136819200 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚22414276069850 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚12089 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119170560 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 225736246 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚83922599 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚30719338047488 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 404597957 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 225752220 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 bMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚1350569790239232 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 116076041 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1186019510171136 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 178845732 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1155300172123648 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400!t6r‘*î+‹U TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15974 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 20369637376 trainer/global_step‚12089 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119170560 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 225736246 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚83922599 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚30719338047488 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 404597957 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 225752220 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2753 bMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚1350569790239232 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 116076041 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1186019510171136 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 1972133464 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 178845732 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1155300172123648 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 17729555456 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 3303014400 [FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚1373019353791488 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 199998640 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 18093162496 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 1950718464 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚839 $ _timestamp‚1.788447848946451e+09 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 20788341248 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 3247569920 \GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚1373021328804352 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚83922468 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 20424465233 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚1575 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24124 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚22451514270720 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1373021328804352 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚22449563552256 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 3245630568 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚30719218876928 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚2649 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚2649 _step‚991 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 178845737 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 199998507 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 225752220 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚1575 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚22449563552256 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581978 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚609 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 414947328 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚22412327267202 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚1575 _runtime‚10288.991510766 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 404597957 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 136819200 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 3247569920 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1074 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚22414276069850ßN4ï+/ _wandbruntime‚10289  _runtime‚10289bo<#—)ð+‘S SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 20424465233 dODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚1328995501674496 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 3245630568 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 23672651776 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 1975012864 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚1575 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 178856626 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24124 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 20369637376 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚1575 _runtime‚10289.958821128 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 aLDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚1350643573121536 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 1975012864 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 3247569920 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 225759434 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 200010603 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚83926632 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1186081829909504 `KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚1350643548827136 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚17648640 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚30719812250624 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 414947328 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 1972133464 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1074 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚132 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚1351408278296770 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 119170560 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚30719693080064 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚2649 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 ]HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚1350643548827136 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚22450037755392 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 20369637376 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 200010469 `KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚1328995478343680 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚22414748755738 trainer/global_step‚12089 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚1351410250430234 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 178856631 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 17729555456 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚22412799953090 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 1948802648 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 404600086 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 3303014400 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15974 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600086 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1074 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 23672651776 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2753 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚2649 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚22451988473856 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚134 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 116083971 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 3303014400 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚1575 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404616065 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2753 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1373093586582528 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24124 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚2649 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 404616065 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 20369637376 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 136819200 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2730509312 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ]HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚1373095561595392 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 178856631 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 116083969 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚22451988473856 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1350643573121536 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1155362154478080 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 225743460 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 % _timestamp‚1.7884478499137423e+09 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 178856626 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 136819200 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚839 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 2730509312 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 3303014400 ]HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚1373095561595392 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 178856631 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 116083969 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 3247569920 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 20788341248 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15974 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚22451988473856 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1438 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚609 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 dODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚1350643573121536 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 1950718464 iTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚1155362154478080 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 2338332672 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1074 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 18093162496 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 225743460 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚2649 % _timestamp‚1.7884478499137423e+09 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 18093162496 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 178856626 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 B&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5771697759628296 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11239143460988998 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17847353219985962 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1953682005405426 5 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16158527135849 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9303194880485535 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8210397362709045 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9970971345901489 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2660919427871704 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4845028519630432 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9845110177993774 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.5100599527359009 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5017809271812439 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42603906989097595 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900158047676086 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5567474365234375 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632002353668 )jet/accuracy_2_of_4‚0.396273672580719 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.031712885946035385 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5222206115722656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46310660243034363 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18067717552185059 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36388885974884033 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5295822024345398 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5663642883300781 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2040853202342987 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14766468107700348 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25329649448394775 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5590278506278992 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3564668297767639 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15782828629016876 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 (particle/accuracy‚0.8494231700897217 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4504357874393463 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01182051282376051 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006920866668224335 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0055644516833126545 particle/accuracy_3_of_4‚1 jet/accuracy_3_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42048949003219604 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6137924790382385 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07473722100257874 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1634093075990677 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970709085464478 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17997418344020844 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.028469491750001907 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9957482814788818 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_7‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.14221711456775665 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42531201243400574 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5590530633926392 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 'particle/f_score‚0.4011859893798828 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2222222089767456 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 *jet/accuracy_2_of_5‚0.5305234789848328 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34241074323654175 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18067717552185059 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911614060401917 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9889892339706421 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1301378756761551 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/2FRt0SRqqt*FBt/event_purity‚0.033278938382864 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5672861933708191 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.41957366466522217 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5605255961418152 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 jet/accuracy_4_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44146212935447693 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18926039338111877 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5140652656555176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4127868413925171 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0436507947742939 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5528575778007507 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1160052940249443 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9816809892654419 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1269092708826065 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21479792892932892 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.568928062915802 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5351998805999756 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5304399132728577 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8333333730697632 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.593380331993103 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.970481276512X‡¶¢Ê8146 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_6‚1 jet/accuracy_7_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941444993019104 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8333333730697632 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18581080436706543 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.533164918422699 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5179322361946106 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6468254327774048 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.32592594623565674 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5218852162361145 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 particle/accuracy_3_of_7‚1 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7291666269302368 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22391536831855774 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.40555545687675476 jet/accuracy_1_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.039007093757390976 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5141990780830383 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0808366984128952 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5812498927116394 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6044761538505554 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7884478632554898e+09 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015847820788621902 particle/accuracy_1_of_5‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886969923973083 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3110514283180237 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2755754590034485 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36424165964126587 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11474957317113876 particle/accuracy_4_of_5‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5278375744819641 +particle/specificity‚0.9468130469322205 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5180236101150513 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12239708006381989 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5055684447288513 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5140652656555176 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10337908565998077 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8734032511711121 ,jet/accuracy_3_of_3‚0.028964824974536896 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4305555522441864 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961973428726196 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5857638120651245 *jet/accuracy_1_of_1‚0.4898030459880829 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2083333283662796 particle/accuracy_4_of_7‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4550109803676605 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13470330834388733 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2812138795852661 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197531044483185 jet/accuracy_5_of_6‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21751388907432556 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_7_of_7‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25563403964042664 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2719592750072479 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333335518836975 jet/accuracy_6_of_7‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4895593523979187 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5712271928787231 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899652004241943 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6041666865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5158231258392334 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6465343832969666 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39188987016677856 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 epoch‚9 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5486453175544739 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5560516119003296 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.553205132484436 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚10303.302159637 +jet/accuracy_4_of_5‚0.01666666753590107 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49898964166641235 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14210526645183563 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1241663247346878 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018031150102615356 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05890804901719093 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5112116932868958 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17242036759853363 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5369685888290405 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_4_of_6‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025307221338152885 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886142611503601 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03742138296365738 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 )loss/val_total_loss‚40.08763620764605 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3960716128349304 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7043904066085815 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04422498494386673 8"Purity/2FRt*SRqqt*FBt/event_purity‚0.013392373919487 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.590175449848175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 particle/accuracy_5_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21469508111476898 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17575287818908691 :"Purity/2FRt1SRqqt*FBt/event_purity‚0.00266400258988142 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14108483493328094 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5361985564231873 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13164913654327393 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06770576536655426 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13690434396266937 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4820345938205719 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21607252955436707 particle/accuracy_5_of_5‚1Ðç (Gö+¬¨ * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5269331336021423 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5453611016273499 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915558099746704 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10684523731470108 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1213991716504097 +jet/accuracy_2_of_6‚0.49358975887298584 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 trainer/global_step‚12089 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6000000238418579 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5253027677536011 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48258429765701294 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5589064955711365 particle/accuracy_2_of_2‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1905330866575241 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3662501275539398 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2107812762260437 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.993831992149353 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10379873961210251 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4689655005931854 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15342146158218384 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2514711022377014 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.363840252161026 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4305555522441864 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ;"Purity/*FRt3SRqqt1FBt/event_purity‚0.018518518656492233 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5771697759628296 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11239143460988998 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17847353219985962 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1953682005405426 5 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16158527135849 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9303194880485535 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8210397362709045 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9970971345901489 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2660919427871704 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4845028519630432 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9845110177993774 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.5100599527359009 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5017809271812439 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42603906989097595 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900158047676086 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5567474365234375 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632002353668 )jet/accuracy_2_of_4‚0.396273672580719 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.031712885946035385 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5222206115722656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46310660243034363 <"Purity/*FRt3SRqqt*FBt/event_purity‚0.0054985336028039455 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18067717552185059 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36388885974884033 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5295822024345398 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5663642883300781 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2040853202342987 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14766468107700348 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.25329649448394775 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5590278506278992 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3564668297767639 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15782828629016876 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 (particle/accuracy‚0.8494231700897217 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4504357874393463 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01182051282376051 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006920866668224335 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0055644516833126545 particle/accuracy_3_of_4‚1 jet/accuracy_3_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42048949003219604 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6137924790382385 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07473722100257874 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1634093075990677 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970709085464478 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17997418344020844 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.028469491750001907 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9957482814788818 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_7‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.14221711456775665 particle/accuracy_6_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42531201243400574 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5590530633926392 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 'particle/f_score‚0.4011859893798828 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2222222089767456 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 *jet/accuracy_2_of_5‚0.5305234789848328 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34241074323654175 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18067717552185059 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911614060401917 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9889892339706421 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1301378756761551 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8"Purity/2FRt0SRqqt*FBt/event_purity‚0.033278938382864 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5672861933708191 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.41957366466522217 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5605255961418152 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 jet/accuracy_4_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44146212935447693 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18926039338111877 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5140652656555176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4127868413925171 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0436507947742939 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5528575778007507 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1160052940249443 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9816809892654419 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1269092708826065 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21479792892932892 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.568928062915802 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5351998805999756 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5304399132728577 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8333333730697632 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.593380331993103 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.970481276512146 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_6‚1 jet/accuracy_7_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941444993019104 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8333333730697632 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18581080436706543 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.533164918422699 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5179322361946106 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6468254327774048 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.32592594623565674 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5218852162361145 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 particle/accuracy_3_of_7‚1 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7291666269302368 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22391536831855774 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.40555545687675476 jet/accuracy_1_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.039007093757390976 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5141990780830383 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.00124007940øEÑ M75801969 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0808366984128952 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5812498927116394 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6044761538505554 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7884478632554898e+09 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015847820788621902 particle/accuracy_1_of_5‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886969923973083 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3110514283180237 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2755754590034485 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36424165964126587 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11474957317113876 particle/accuracy_4_of_5‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5278375744819641 +particle/specificity‚0.9468130469322205 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5180236101150513 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12239708006381989 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5055684447288513 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5140652656555176 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10337908565998077 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8734032511711121 ,jet/accuracy_3_of_3‚0.028964824974536896 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4305555522441864 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961973428726196 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5857638120651245 *jet/accuracy_1_of_1‚0.4898030459880829 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2083333283662796 particle/accuracy_4_of_7‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4550109803676605 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13470330834388733 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2812138795852661 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197531044483185 jet/accuracy_5_of_6‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21751388907432556 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_7_of_7‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25563403964042664 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2719592750072479 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333335518836975 jet/accuracy_6_of_7‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4895593523979187 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5712271928787231 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899652004241943 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6041666865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5158231258392334 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6465343832969666 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39188987016677856 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 epoch‚9 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5486453175544739 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5560516119003296 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.553205132484436 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚10303.302159637 +jet/accuracy_4_of_5‚0.01666666753590107 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49898964166641235 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14210526645183563 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5087719559669495 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1241663247346878 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018031150102615356 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05890804901719093 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5112116932868958 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17242036759853363 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5369685888290405 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_4_of_6‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025307221338152885 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886142611503601 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03742138296365738 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 )loss/val_total_loss‚40.08763620764605 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3960716128349304 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7043904066085815 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04422498494386673 8"Purity/2FRt*SRqqt*FBt/event_purity‚0.013392373919487 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.590175449848175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 particle/accuracy_5_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21469508111476898 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17575287818908691 :"Purity/2FRt1SRqqt*FBt/event_purity‚0.00266400258988142 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14108483493328094 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5361985564231873 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13164913654327393 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06770576536655426 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13690434396266937 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4820345938205719 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21607252955436707 particle/accuracy_5_of_5‚1 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_1_of_7‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29055076837539673 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9867132306098938 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3178931474685669 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_4‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2692507207393646 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9823284149169922 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4879949688911438 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5310898423194885 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12452107667922974 particle/accuracy_1_of_1‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7291666269302368 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_6‚0.1538461595773697 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt3SRqqt1FBt/event_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909427762031555 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21264903247356415 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4207426905632019 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10851223766803741 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9926454424858093 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5338113903999329 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3596161901950836 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1706380695104599 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30051442980766296 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7548254132270813 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015211639925837517 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.010457516647875309 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,particle/sensitivity‚0.32324233651161194 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1508496105670929 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996760487556458 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5937354564666748 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12715473771095276 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.040910810232162476 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5298357605934143 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07597461342811584 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.004464285913854837 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5174512267112732 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5061306953430176 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4633333086967468 _step‚993 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874865412712097 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933807253837585 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.14885057508945465 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25081443786621094 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064ást R÷+ ioojqu7mj@ ÷˜æÔøÅÌ0`Trainer.fit` stopped: `max_epochs=10` reached. ”bœø+‚(:‘ ø˜æÔáöÈ gpu.0.gpu‚6.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚28.198318481445312* gpu.0.memoryAllocatedBytes‚ 24222171136 gpu.0.temp‚31.0 gpu.0.powerWatts‚76.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.274799999999999 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­¹»X:ù+‚(:¯ ø˜æÔª‘ßË proc.cpu.threads‚17 disk.rbd1.in‚0" proc.memory.rssMB‚ 4292.0390625 disk.sda2.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚84196.26513671875 cpu‚25.342158506723013# disk.nvme0c0n1.in‚ 1778.26171875( disk.nvme0c0n1.out‚84196.26513671875 network.recv‚ 3.710019e+06 disk.drbd1777.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 6.2578125 disk.dm-1.in‚ 0.57421875 disk.sda1.in‚0% disk./.usageGB‚1108.3440322875977 disk.rbd3.out‚0! disk.nvme0n1.in‚ 1778.26171875$ memory_percent‚99.96180534362793 disk.sda.out‚687.125 disk.rbd0.in‚0 disk.dm-0.in‚ 1778.26171875 disk.rbd3.in‚0 disk.sda3.out‚687.125) disk./.usagePercent‚74.38189796940559 disk.rbd1.out‚ 5.275390625 disk.rbd2.in‚0 disk.sda.in‚ 0.57421875 disk.sda2.in‚0( proc.memory.percent‚26.1965274810791 network.sent‚ 3.0668226e+07 disk.dm-1.out‚687.125 disk.drbd1777.out‚0 disk.rbd2.out‚ 672.1640625& disk.nvme0n1.out‚84196.26513671875 disk.sda3.in‚ 0.57421875Îe6Ÿú+ ioojqu7mjŒ ù˜æÔ˜¦š…zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:15:50 • 0:00:00 1.27it/s v_num: qu7my® %û+ ioojqu7mj ù˜æÔˆ–¯…  L4ü+/ _wandbruntime‚10307  _runtime‚10307ŒzÜãý+‚(’ÃP